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ABSTRACT 


This  research  develops  and  demonstrates  the  use  of  a  Preference-Characteristic 
methodology  to  select  between  scenarios  for  the  purpose  of  training  a  unit  in  a  virtual  or 
constructive  environment.  The  methodology  proved  successful  though  additional 
research  will  be  required  before  the  tool  can  become  fully  operational. 

The  effort  to  exploit  the  potential  of  simulation  for  improving  task  proficiency 
began  in  the  early  1970’s.  Technology  at  the  time  limited  most  efforts  to  improvements 
in  live  simulation  training.  In  1973  the  Constructive  Tactical  Engagement  System  (TES) 
was  the  first  Army  organizational  computer  based  simulation  training  device. 

The  advancement  in  technology  and  the  successful  use  of  these  initial  training 
tools  lead  to  the  expansion  of  simulation  training  devices.  Along  with  the  explosion  of 
computer  based  simulations  was  an  order  of  magnitude  explosion  in  the  number  of 
training  scenarios.  The  number  of  scenarios  became  so  unwieldy  that  numerous 
governmental  review  boards  recommended  that  efforts  be  made  to  catalog  them  and 
place  them  in  libraries. 

To  solve  this  problem  the  Army  developed  exercise  development  tools  to  serve 
the  diverse  training  population.  This  research  effort  refines  that  process.  The  geographic 
dispersion  of  the  users  and  limited  resources  mandated  the  use  of  a  survey  as  a  collection 
tool.  The  survey  was  conducted  in  two  phases:  1)  an  informal  pilot  phase  used  to 
develop  and  refine  the  survey  instrument  along  with  analysis  techniques  and  2)  a  formal 
pilot  test  where  members  of  the  targeted  populations  were  solicited  for  their  responses. 


Two  types  of  statistical  analysis  were  applied  to  the  data,  mean  based  and  frequency 
based.  The  outcome  of  the  analyses  was  a  weight  matrix  that  quantified  the  impact  each 
selection  factor  imposed  on  the  decision.  The  quantified  data  can  then  be  applied 
through  a  mathematical  model  that  uses  these  weights  to  differentiate  between  previously 
equivalently  ranked  alternatives.  An  automated  search  was  accomplished  by  quantifying 
the  characteristics  of  the  scenarios  and  the  users  preferences  against  the  same  criteria. 
Simple  mathematics  and  matrix  comparisons  and  calculations  were  then  be  used  to 
quantify  the  level  of  agreement  between  the  scenarios  and  the  users  preferences.  An 
automated  example  of  the  selection  methodology  demonstrates  and  yields  a  correct 
solution. 

The  methodology  developed  in  the  study  can  be  applied  to  any  scenario  selection 
process.  Only  the  codification  and  knowledge  acquisition  are  domain  dependent. 
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CHAPTER  1  IMPROVING  RESPONSIVENESS  OF  SIMULATION  TOOLS 


Introduction 


“Everything  is  simulation  except  combat.” 

Defense  Science  Board,  1993 


All  skill  training  that  takes  place  apart  from  the  environment  in  which  it  will  be 
utilized  is  simulation.  The  purpose  of  the  training  exercise  is  not  important.  A  salesman 
rehearsing  his  pitch  about  a  new  car  model  and  a  tank  gunner  engaging  targets  in  a 
virtual  environment  are  both  honing  their  skills  in  an  effort  to  improve  their  levels  of 
performance.  They  can  improve  their  level  of  proficiency  through  the  repetition  of  their 
tasks  in  an  environment  that  supplies  the  same  cues  as  the  real  world.  Both  are  training 
with  simulation. 


“Large  scale  maneuvers  are  a  thing  of  the  past....  The  ecologists 
will  make  you  stop  fighting  if  you  raise  too  much  dust.” 

General  W.  E.  Depuy,  1973 
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Recognizing  the  impact  of  mounted  maneuver  exercises  on  the  environment, 
General  Depuy’s  made  this  statement  as  the  U.S.  Army  began  a  massive  drawdown  in 
strength.  These  comments  were  not  made  after  the  fall  of  the  Berlin  Wall  in  the  late 
1980’s  but  in  1973  as  the  United  States  moved  from  a  conscription  military  to  an  all 
volunteer  force  (Gorman,  The  Secret  of  Future  Victories,  1992).  As  the  Chief  of  Staff 
of  the  Army  in  1971,  General  Westmoreland  had  created  the  Board  For  Dynamic 
Training  (BFDT)  (Gorman,  1992).  Westmoreland  established  the  BFDT  with  the 
mandate  to  design  realistic  military  training  that  compensated  for  the  shortages  of  firing 
and  maneuver  ranges  and  ammunition  while  maximizing  soldiers’  safety  and  minimizing 
cost  and  environmental  impact.  The  BFDT  accomplished  its  mission  by  designing 
Tactical  Engagement  Simulations  (TES)  which  supported  maneuver  training  and  battle 
simulations  which  supported  the  decision  making  process. 

BFDT’s  successor,  the  Combat  Arms  Training  Board,  developed  TES  systems  for 
live  simulation  training.  The  first  successful  fielded  systems  were  SCOPES  (not  an 
acronym)  and  REALTRAIN  (not  an  acronym)  (Orlansky,  The  Value  of  Simulation 
Training,  1994).  SCOPES  used  a  rifle  mounted  optical  sight  and  a  number  attached  to 
each  soldier’s  helmet.  The  system  assessed  a  “kill”  when  the  firing  soldier  could  read  the 
target’s  number.  REALTRAIN  was  a  similar  system  developed  for  mounted  combat 
training.  These  systems  led  to  the  development  of  the  Multiple  Integrated  Laser 
Engagement  System  (MILES)  which  was  distributed  throughout  the  Army  in  the  early 
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1980’s  as  the  primary  training  device  for  the  simulations  conducted  at  the  Combat 
Training  Centers  (CTC’s). 

The  constructive  simulations  did  not  really  advance  until  the  1 973  Arab-Israeli 
war  (Gorman,  The  Military  Value  of  Training,  1990).  The  Army  studied  how  Israel 
fought  outnumbered  and  won.  This  battle  simulation,  constructive  TES,  was  the 
forerunner  of  the  computer  simulations  used  for  analysis  and  command  post  exercises 
today. 

As  the  commander  of  The  U.S.  Army’s  Training  and  Doctrine  Command 
(TRADOC),  General  Depuy  asserted  that  simulation-based  training  (primarily  TES)  was 
important  in  that  it  trained  the  soldiers  to  function  as  teams.  The  gains  achieved  in 
teamwork  and  synchronization  were  far  more  important  than  improvement  in  individual 
skills.  The  U.S.  Army  currently  recognizes  that  training  in  simulation  benefits  both 
performance  and  “the  evaluation  of  concepts,  the  evaluation  of  weapon  systems,  supply 
[strategies],  tactics  and  doctrine.”  (Orlansky,  1994) 


Simulation  Training  Today 

Although  some  overlap  does  occur,  the  U.S.  Army  groups  simulation  training 
into  four  categories:  live  simulation,  virtual  simulation,  constructive  simulation  and 
distributed  simulation. 
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Live  simulation  involves  operations  with  actual  units  and  equipment  in  real 


space  under  as  close  to  battle  conditions  as  possible  (Piplani  et  al,  Systems  Acquisition 
Manager’s  Guide  for  the  Use  of  Models  and  Simulations.  19941.  The  training  location 
maximizes  the  soldiers’  ability  to  practice  the  tactics,  techniques,  and  procedures  that 
will  ensure  unit  success  in  combat.  The  soldiers’  actions  are  monitored,  adjudicated,  and 
evaluated  using  the  tools  of  live  simulation.  Systems  such  as  the  Multiple  Integrated 
Laser  Engagement  System  (MILES),  the  Tank  Weapons  Gunnery  Simulation  System 
(TWGSS),  and  instrumented  ranges  monitor  the  unit’s  progress  and  provide  feedback. 

Like  live  simulations,  virtual  simulations  immerse  individuals  and  crews  in  a 
realistic  training  environments  designed  to  replicate  combat.  Virtual  simulation  differs  in 
that  first  and  foremost  the  environment  is  a  computer  generated  or  enhanced  battlefield 
that  is  sensed  or  visualized  by  the  trainee.  The  trainee  then  interacts  with  this 
environment  via  high  fidelity  representations  of  equipment.  The  simulation  attempts  to 
replicate  the  trainees’  (usually  a  crew  of  a  combat  vehicle  or  weapon  system)  equipment 
and  external  environment  as  much  as  possible.  Crew  actions  cause  the  simulator  to 
interact  with  its  virtual  world  as  the  real  vehicle  would  interact  in  the  real  world.  Thus, 
the  crew  can  train  on  individual  and  collective  tasks  as  well  as  battlefield 
synchronization.  The  best  examples  of  virtual  trainers  are  the  Ml  Tank  Driver  Trainer, 
the  Unit  Conduct  of  Fire  Trainer  (UCOFT),  and  aircraft  pilot  trainers. 

Constructive  simulations  are  complex,  deterministic  and  stochastic  computer 
driven  models  used  to  train  commanders  and  their  staffs  to  monitor,  evaluate  and  develop 
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the  battle.  In  a  constructive  simulation  the  battle  space  and  the  player  entities  exist 
entirely  within  the  computer.  The  primary  audience  of  trainees  are  usually  isolated  from 
this  constructed  computer  simulation  world  by  organic  decision  support  systems. 
Maneuver,  attrition,  and  supply  algorithms  simulate  the  actions  of  real  units  in  the 
training.  Computer  operators  interact  with  the  constructive  simulation’s  computer 
interfaces  to  generate  the  reports  that  track  the  flow  of  the  battle.  Some  of  the  most 
widely  used  constructive  simulations  in  the  Army  are  Janus,  Battalion/Brigade  Battle 
Simulation  (BBS),  and  the  Corps  Battle  Simulation  (CBS). 

Blurring  the  boundaries  between  live,  virtual,  and  constructive  simulation, 
Distributed  Interactive  Simulations  (DIS)  facilitate  the  linking  of  all  forms  of  simulation. 
This  linking  can  occur  at  any  scale.  Examples  are  the  integration  of  two  individual 
training  simulations  located  in  the  same  room  connected  via  a  local  area  network  (LAN) 
or  the  linking  of  large  units  geographically-dispersed  over  a  wide  area  using  dedicated 
communications  linkages  to  train  higher  level  staffs  (Hollenbach,  1995).  In  the 
distributed  simulations,  constructive  entities  can  interact  with  manned  virtual  entities. 
Live  entities  operate  within  their  own  environment,  but  can  be  acted  upon  by  virtual  or 
constructive  entities.  The  integration  of  different  simulations  allows  the  scaling  of  the 
training  audience.  Individuals,  teams,  and  units  can  all  interact  in  the  same  battle  space 
through  live,  virtual,  and  constructive  means.  The  ability  to  integrate  across  simulation 
environments  and  certain  simulators  is  very  dependent  on  the  architecture  used  to 
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transfer  and  trace  data  in  the  form  of  Protocol  Data  Units  (PDU’s)  in  the  simulations 
(Brann,  1995)  (Hollenbach,  1995). 

The  Simulation  Networking  Trainer  (SIMNET)  and  the  Combined  Arms  Tactical 
Trainer  (CATT)  family  are  examples  of  distributed  simulations  whose  interface  and 
distributed  nature  is  so  transparent  to  the  trainee  that  the  simulation  appears  to  be  virtual. 
The  immersive  virtual  nature  of  the  ihdividual  simulators  causes  the  users  to  consider  the 
trainers  to  be  virtual  simulator  systems. 

This  paper  seizes  on  the  overwhelming  virtual  appearance  these  simulators 
present  to  their  training  audience  and  includes  them  when  referring  to  virtual  simulations. 

Training 

The  U.S.  Army’s  simulation  programs  are  integrated  into  its  overall  training  plan. 
All  training  in  the  Army  begins  with  an  assessment  of  the  unit’s  proficiency  based  on  the 
unit’s  Mission  Essential  Task  List  (Headquarters  the  Department  of  the  Army  (HQDA), 
1990).  The  Mission  Essential  Task  List  (METL)  is  a  list  of  the  critical  wartime 
functions  for  the  unit.  Each  unit  is  rated  on  each  task  in  terms  of  being  trained,  needing 
practice,  and  being  untrained.  This  process  gives  the  commander  a  list  of  tasks  on  which 
his  unit  must  either  sustain  or  improve  their  performance. 

The  unit  commander  and  staff  then  incorporate  the  higher  commander’s  guidance 
along  with  the  status  and  availability  of  several  factors  (e.g.,  training  audience,  training 
objectives,  risk  assessment,  and  available  resources).  In  simulation-based  training,  the 
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commander  must  also  assess  the  availability  of  Training  Aids,  Devices,  Simulators  and 
Systems  (TADSS)  and  then  determine  which  training  scenarios  for  the  particular 
simulation  system  would  best  support  the  execution  of  the  training  (HQDA, ,  1996).  The 
commander  ensures  that  the  training  scenario  replicates  the  appropriate  conditions  and 
standards  that  the  unit  will  encounter  in  combat  (U.S.  Army  Research  Institute  (ARI), 
1993). 

The  Effects  of  Virtual  and  Constructive  Simulation  Training  in  the  U.S.  Army 
The  Army  has  garnered  great  improvements  in  its  operational  readiness  from  the 
inclusion  of  virtual  and  constructive  simulation  based  training.  Simulator-based  training 
provides: 

•  Excellent  immediate  or  short  delay  feedback 

•  The  opportunity  to  repeat  situations  that  were  not  completed  successfully 

•  The  flexibility  to  modify  training  situations 

•  Great  reduction  in  resource  expenditure  and  risk  (Orlansky,  1994). 

The  trainee  receives  prompt  and  accurate  feedback  because  of  the  computer  based 
nature  of  virtual  and  constructive  simulations.  The  crew  manning  a  tank  in  a  virtual 
battle  receives  feedback  on  their  performance  as  they  maneuver.  If  they  receive  enemy 
fire,  they  learn  that  they  exposed  themselves;  if  they  fail  to  maintain  their  place  in 
formation  they  can  view  this  problem  when  the  battle  is  replayed.  Similarly,  constructive 
simulations  show  a  commander  if  the  staff  is  capable  of  monitoring  and  interpreting  the 
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course  of  the  battle.  Any  poor  decisions  could  result  in  abnormal  loss  ratios,  untenable 
positions,  or  a  commander’s  inaccurate  perception  of  the  battle. 

Simulations  offer  the  commander  the  ability  to  repeat  tasks  and  scenarios  that 
their  forces  have  failed  to  master.  If  a  unit  makes  a  drastic  mistake  the  trainer  can  simply 
reset  the  scenario  to  the  start  time.  This  allows  the  unit  to  learn  from  its  mistakes  while 
their  errors  are  still  fresh.  Repeatability  and  immediate  feedback  are  two  of  the  most 
important  keys  to  learning  (Anderson,  Rules  of  the  Mind.  1993  ).  In  live  simulations,  the 
process  of  moving  troops  to  their  original  locations  could  require  lots  of  time.  The  re¬ 
setting  of  the  live  training  scenario  consumes  valuable  training  time  and  resources;  this 
consumption  of  time  and  resources  takes  away  from  the  unit’s  opportunity  to  reinforce 
training. 

The  money  saved  in  simulation  training  comes  from  three  sources:  the  reduction 
in  the  use  of  vehicles,  the  savings  associated  with  the  repair  of  maneuver  damage,  and  the 
simulator’s  life  cycle  costs  (HQDA,  Battle  Focused  Training.  1990).  Some  of  the  most 
significant  savings  occur  in  vehicle  operation  training  simulators  as  indicated  in  Table  1. 

TABLE  1  INCREMENTAL  DIFFERENTIAL  COST 


Weapon  System 

Simulator 

Cost/(Unit) 

System  Cost/(Unit) 

M1A1 

$2. 5  0/mile 

$75.00/mile 

UH-60 

$59.00/hour 

$  1,448/hour 

AH-64 

$70.00/hour 

$3, 101/hour 

(National  Training  Systems  Association,  1997) 
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Huge  savings  can  also  be  attained  by  applying  virtual  or  constructive  technologies 
to  live  simulations.  In  1992  constructive  and  virtual  constructs  were  applied  to  the 
annual  Return  of  Forces  to  Germany  (REFORGER)  exercise.  $34  million  were  saved  as 
compared  to  an  equivalent  exercise  that  occurred  in  1988  (ADPA,  1997). 

There  are  also  great  costs  associated  with  deploying  maneuver  forces  to  the 
training  location.  The  excessive  costs  in  terms  of  money,  time  and  resources  of  moving 
equipment  to  large  training  areas  by  truck,  rail  or  barge  can  consume  training  budgets. 
U.S.  Army  Combat  Training  Center  Directorate  breaks  down  of  costs  of  a  typical 
National  Training  Center  (NTC)  rotation  of  a  Brigade  Combat  Team  consisting  of  two 
Battalion  Task  Forces  from  Fort  Riley  Kansas  follows  in  Table  2. 

TABLE  2  NTC  ROTATION  BUDGET 


Budget  Item 

Cost 

Transportation  of  Personnel  (4,  000) 

$1.6  million 

Transportation  of  equipment  (Barge) 

$800  thousand 

Logistical  base  and  supplies 

$2.3  million 

Training  Cycle 

$1.3  million 

Total 

$6  million 

(Croteau,  Telephone  Interview,  1997) 

In  this  example,  21%  of  the  exercise  budget  is  expended  during  the  conduct  of 
training.  The  majority  of  the  training  funds  are  spent  gathering  the  players,  their 
equipment,  and  the  resources  that  will  be  consumed  during  the  exercise.  It  is  often 
difficult  to  coordinate  units’  arrivals.  This  leads  to  the  consumption  of  additional 
resources. 
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It  is  also  expensive  to  train  and  equip  an  Opposing  Force  (OPFOR).  Beyond  the 
costs  associated  with  equipping  the  OPFOR  with  equipment  that  accurately  depicts  the 
form  and  capabilities  of  the  threats  weapons,  one  also  has  to  support  the  infrastructure  of 
an  entire  brigade  that  does  not  contribute  to  the  Army’s  combat  projection. 

The  savings  in  resources  extends  beyond  funds.  Simulation  training  compensates 
for  limits  on  maneuver  space,  the  gathering  of  forces,  a  lack  of  a  trained  OPFOR,  and  the 
inability  to  replicate  the  logistical  battle  (HQDA,  Battle  Focused  Training,  1990).  As 
General  Depuy  indicated  in  the  early  1970’s,  the  Army  has  limited  land  available  for 
large  maneuver  forces.  The  National  Training  Center,  at  Fort  Irwin,  California  is  one  of 
the  few  remaining  training  areas  that  can  support  brigade  level  live  fire  maneuver 
exercises.  The  majority  of  training  areas  can  support  only  restricted  battalion  level 
operations. 

Shortfalls  of  Simulation  Training 

In  an  effort  to  improve  the  quality  of  simulation  based  training,  the  U.S.  Army 
has  commissioned  26  reviews  of  its  training  simulation  programs  since  the  1970’s 
(Flectcher,  A  Review  of  Study  Panel  Recommendations  For  Defense  Modeling  and 
Simulation,  1992).  Many  of  the  recommendations  focus  on  two  areas  concerning 
training  scenario  selections.  Many  of  the  boards  recommended  that  the  Army  reduce  the 
planning  and  support  requirements  needed  to  conduct  simulation  training  and  that 
libraries  be  established  from  which  commanders  can  choose  training  scenarios. 
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The  1988  Report  of  the  Defense  Science  Board  Task  Force  on  Computer 
Applications  to  Training  and  Wargaming  (AD-A199  456)  and  the  1989  Wargaming 
Activities  in  the  Department  of  Defense  (Report  NO.  89-057)  both  recommended 
reducing  the  burden  associated  with  planning  simulation  based  training  (establishing 
scenarios,  placement  of  forces,  etc.).  These  studies  found  that  there  was  a  great  deal  of 
overhead  associated  with  conducting  training,  particularly  with  Reserve  Component 
(RC)  units.  The  study  stated  that  the  benefit  of  the  training  is  impacted  by  planning 
requirements  and  that  the  units’  readiness  would  improve  if  the  planning  burden  was 
reduced. 

Several  review  boards  have  also  recommended  the  establishment  of  data  or 
archived  file  libraries  to  hold  simulation  data  and  exercises  for  use  by  the  troops.  The 
Army  Science  Board  Final  Report  of  the  Ad  Hoc  Subgroup  on  the  use  of  Army  Combat 
Models  for  the  Analysis  and  Training  of  Joint  Combined  Operations  (AD-B120  937) 
and  the  1989  Wargaming  Activities  in  the  Department  of  Defense  both  made  this 
recommendation.  The  Army  Science  Board  1991  Summer  Study  on  Army  Simulation 
Strategy  (Draft)  further  recommended  that  a  catalog  of  certified  and  uncertified 
simulations  and  scenarios  be  established. 

Other  recommendations  for  changes  include  the  1988  Report  of  the  Defense 
Science  Board  Task  Force  on  Computer  Applications  to  Training  (AD-A199  456)  and 
Wargaming’s  desire  to  integrate  allied  forces  into  scenarios  so  that  U.S.  commanders  can 
train  for  multi-national  operations.  Dr.  Larocque,  Director  of  the  National  Simulation 
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Center  at  Ft.  Leavenworth,  Kansas,  and  the  101st  Airborne  Division’s  (Air  Assault) 
Division  Training  Chief,  Major  Michael  Rossi,  both  assert  that  work  must  be  done  to 
make  the  tools  associated  with  planning  simulation  training  more  responsive  to 
commanders.  Both  recommended  that  the  selection  should  be  based  on  mission,  enemy, 
time,  terrain,  and  troops  (METT-T)  considerations  (Department  of  the  Army,  ARTEP  7- 
30-MTP,  1989). 

A  unit’s  METL  is  the  Mission  Essential  Task  List  that  is  composed  of  the  most 
important  objectives  the  unit  must  be  able  to  accomplish  during  war  (Combined  Arms 
and  Service  Staff  School,  Training  Management.  1991).  A  sample  METL  for  an  infantry 
company  (C  company  1st  battalion  -27th  Infantry  Regiment  (Wolfhounds)  is  composed 
of:  Deploy,  Assault,  Move  Tactically,  Perform  Reconnaissance,  Defend,  Occupy  an 
Assembly  Area,  Perform  Air  Assault  Conduct  Non-combatant  Evacuation  Operations, 
Infiltrate/  Exfiltrate,  Consolidate  and  Reorganize  (Gubler,  Personal  Interview,  1996). 
Where  the  METL  are  the  activities,  METT-T  are  the  conditions  under  which  they  are 
performed. 


12 


CHAPTER  2  PLANNING  TOOLS  AND  A  METHODOLOGY  FOR 
IDENTIFYING  COMMANDERS  NEEDS 

Current  Approaches  to  Planning  Training 
Live  Simulation 

The  majority  of  the  collective  training  experience  of  the  U.S.  Army  over  the  last 
200  years  has  focused  on  live  simulation.  All  of  the  training  the  U.S.  Army  conducted 
prior  to  the  1973  Arab-Israeli  War  were  live  simulations  of  combat.  All  training  from 
the  Continental  Army’s  drills  on  the  parade  field  during  the  Revolutionary  War  through 
the  integration  of  SCOPES  and  REALTRAIN  in  the  1970’s  was  live  simulation.  The 
U.S.  Army’s  institutional  training  for  the  entire  period  was  focused  on  maneuvering  live 
forces  on  terrain  replicating  a  real  battlefield.  This  experience  has  led  to  a  highly 
detailed  Training  Management  planning  process  to  gain  the  greatest  benefit  from  the 
exercises. 

Live  simulations  are  also  easily  conceptualized  by  the  training  planners  because 
they  require  concrete  assets  on  land  forms  that  are  familiar  to  them.  The  training 
planners’  life  and  work  experience  are  in  three-dimensional  space.  The  wartime 
execution  of  the  trained  tasks  is  also  going  to  be  in  three  dimensional  space  (real  world  or 
real  space).  This  is  an  important  correlation  for  how  the  trainers  visualize  the  training. 

The  military  training  planner  functions  within  a  formalized  system  of  training 
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cycles  (Combined  Arms  and  Service  Staff  School,  Training  Management.  1991).  Each 
U.S.  Army  battalion  and  brigade  executes  three  training  phases  per  quarter:  post  support, 
mission  readiness,  and  prime  training.  The  phase  with  the  least  amount  of  training  is  one 
of  post  support.  During  post  support  the  unit’s  soldiers  are  used  to  support  the  other 
activities  associated  with  the  installation.  This  ranges  from  guarding  ammunition  storage 
facilities  to  supporting  other  unit’s  training  activities.  Post  support  is  followed  by  a 
period  of  increased  individual  skill,  weapon,  and  small  unit  training.  During  this  period 
the  unit’s  soldiers  are  trained  and  qualified  with  their  weapons  systems  and  the  individual 
skills  associated  with  their  military  occupational  specialty.  The  unit’s  primary  focus  is 
on  its  ability  to  deploy  and  execute  its  war  time  mission.  The  final  phase  consists  of 
rigorous  collective  task  training.  During  this  prime  training  phase  the  individual  skills 
are  integrated  and  synchronized.  This  often  culminates  in  the  evaluation  of  the  unit  at 
company,  battalion,  or  brigade  level. 

This  training  phase  structure  affords  the  training  planner  the  opportunity  to 
conduct  a  leaders  reconnaissance  of  training  sites.  At  the  site,  the  qualified  trainer 
envisions  the  locations  of  units,  their  relative  positions,  and  the  movements  of  the 
elements  during  execution.  Here  the  planners  make  decisions  based  on  their  experience 
and  the  training  area. 

This  process  involves  the  use  of  a  multi-  dimensional  planar  representation  of  the 
training  area-a  map.  The  map  allows  the  planners  to  understand  the  relative  positions  of 
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the  training  areas.  The  map’s  depiction  of  the  actual  land  forms  is  somewhat  distorted. 
This  distortion  is  based  on  scale  and  time.  The  scale  distortion  results  from  the 
compression  of  multi-dimensional  topographical  and  cultural  features  onto  a  smaller, 
lesser  dimensional  tool.  Decisions  are  made  about  the  importance  of  objects  based  on 
their  magnitude.  Topographical  features  under  selected  sizes  cannot  be  represented. 
Though  this  information  may  be  essential  at  the  tactical  level,  it  may  not  be  extractable 
from  the  map. 

As  a  static  representation  of  the  terrain,  a  map’s  data  does  not  reflect  changes 
made  since  publication.  Over  time,  the  natural  effects  of  erosion  and  man’s  impact  on  the 
terrain  distance  reality  from  the  original  representation.  Natural  events  and  man’s 
actions  render  maps  increasingly  more  inaccurate  as  time  proceeds. 

The  impact  of  these  distortions  decreases  as  the  primary  training  audience  shifts 
from  the  company  level  to  battalion  and  above  .  The  importance  of  the  placement  of 
minor  land  forms  at  a  unit  or*  individual  level  decreases  as  the  exercise  includes  larger 
groups.  The  impact  is  still  felt  locally  by  the  elements  involved  at  the  site,  but  that 
impact  is  diluted  at  the  larger  scale. 

The  coordination  of  training  resources  and  assets  is  also  more  easily  understood 
in  live  training.  The  correct  placement  of  assets  and  their  need  is  most  demonstrateable 
when  their  impact  is  visualized.  The  difficulty  in  maneuvering  a  tank  that  is  equipped 
with  a  mine  clearing  plow  over  wooded  terrain  is  more  clearly  understood  when  seen 
than  when  one  simply  acknowledges  a  reduced  movement  speed.  The  same  effect  is 
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noted  when  an  event  cannot  take  place  because  needed  equipment  is  not  in-place  when 
required.  The  impact  of  trainees  failing  in  their  exercise  is  greater  if  the  impact  is  direct 
and  in  real-time. 

The  impact  in  live  simulation  can  be  seen  during  a  training  exercise  at  the  local 
training  areas  around  any  Army  installation.  A  battalion  task  force  training  on 
defensive  operations  must  notify  its  associated  engineer  company  of  what  training  tasks  it 
wants  to  accomplish.  The  engineer  assets  are  then  moved  to  the  training  area.  At  the 
training  area,  the  engineers  are  directed  to  help  prepare  defensive  fortifications,  lay  mines 
and  create  obstacles.  A  properly  prepared  defense  may  take  72  or  more  hours  to 
emplace.  The  defensive  position  is  then  defended. 

At  the  end  of  the  defense,  the  task  force  commander  must  decide  if  the  unit  needs 
to  repeat  any  portion  of  the  exercise,  if  another  element  must  complete  the  same  training, 
or  if  the  exercise  is  completed. 

If  the  unit  needs  to  retrain  on  any  of  the  defensive  tasks  it  may  be  necessary  to 
repeat  the  exercise.  This  could  be  accomplished  on  the  same  terrain  or  in  another 
location.  If  either  of  these  options  are  selected,  the  area  of  the  defense  must  be  returned 
to  the  its  original  condition  in  order  to  correct  the  problems  that  occurred  in  the  last 
exercise  or  to  make  ready  to  move  to  a  new  area.  If  the  retraining  is  to  occur  at  the  same 
location  some  training  value  is  lost  because  the  subjects  do  not  review  all  of  the  same 
tasks.  All  of  these  options  involve  great  costs. 
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The  physicality  of  the  landforms  and  assets  used  in  planning  training  in  live 
simulation  also  limit  their  use  versus  the  virtual  and  constructive  arenas.  If  an  asset  is 
planned  to  be  used  in  a  live  simulation  exercise,  it  must  first  exist  and  be  moved  to  the 
proper  location.  Once  in  place,  the  unit,  or  resource,  will  consume  other  resources  in  the 
accomplishment  of  its  tasks.  These  tasks  will  usually  have  impact  on  the  other  units  in 
the  area  and  the  landform.  This  impact  means  that  if  the  asset  is  to  be  used  again  some 
effort  must  be  made  to  return  it  to  its  initial  conditions  and  move  it  or  the  elements 
around  it  for  its  next  use.  This  requires  a  great  deal  of  effort  in  planning  to  accomplish 
efficiently. 

Virtual  and  constructive  simulations  do  not  have  this  physical  impact  problem. 
Resources  and  units  can  be  created  exclusively  for  the  exercise.  Fuel  for  live  exercises 
must  be  purchased  and  distributed  to  the  forces  for  consumption.  In  virtual  exercises  the 
resource  can  simply  be  incremented  from  a  controller’s  workstation.  Unit  strengths  can 
also  be  treated  this  way.  In  virtual  or  constructive  simulation  an  enemy  force  can  be 
automated  or  semi  automated  (SAF).  This  allows  the  trainers  to  conserve  resources 
operating  the  enemy  as  opposed  to  live  training  where  Opposing  Forces  (OPFOR)  need 
all  the  same  resources  as  the  primary  training  audience. 

In  virtual  and  constructive  simulation  the  transition  to  the  next  training  task  is 
almost  effortless.  Virtual  forces  can  be  relocated  to  any  place  in  the  computer  based 
battlefield  in  a  moment  of  time  with  the  effort  of  a  skilled  computer  operator.  The 
transition  also  does  not  involve  returning  the  training  area  to  its  original  condition.  The 
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units  do  not  have  to  fill  in  dug  in  positions,  retrieve  sensors  and  other  equipment,  and 
reduce  obstacles  that  were  deployed  during  the  exercise. 

Thus,  the  factors  that  make  planning  live  training  easier  to  conceptualize  become 
constraints  in  that  expensive  resources,  detailed  planning,  and  large  amounts  of 
coordination  have  to  be  accomplished  in  order  to  provide  worthwhile  training  to  any  unit 
larger  than  squad  size. 


Virtual  and  Constructive  Training 

Planning  virtual  and  constructive  training  is  more  difficult  than  planning  live 
training  because  of  the  lack  of  a  physical  interface.  The  inability  to  see  and  understand 
the  training  area  reduces  the  planner’s  ability  to  predict  the  direct  actions  of  the  units. 
This  inability  to  conceptualize  the  flow  of  actions  and  predict  the  actions  of  units  at  the 
lowest  level  alters- the  training  at  the  higher  levels. 

A  trainer  working  with  virtual  and  constructive  simulation  also  has  many  more 
choices  than  trainers  who  use  live  simulation.  The  large  costs  associated  with  the 
movement  and  operation  of  people  and  equipment  over  great  distances  limits  most  live 
training  to  areas  close  to  the  element’s  permanent  location.  Virtual  and  constructive 
training  are  not  similarly  limited.  Once  the  virtual  and  constructive  simulator  training 
sites  are  established  (physically  established  training  center,  not  digitized  terrain),  units 
can  use  these  facilities  to  train  in  any  location  available  to  the  database  (as  represented  in 
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the  virtual  or  constructive  world).  The  training  planner  can  conduct  training  under 
almost  any  conditions  at  any  location  and  move  from  one  position  to  the  other  with 
essentially  no  significant  impact.  This  escalation  in  the  number  of  training  options 
without  a  significant  change  in  planning  time  leads  to  a  less  thoughtful  consideration  of 
each  option.  As  the  number  of  options  (scenarios)  increases  while  the  planning  time 
remains  constant,  the  consideration  time  per  option  decreases.  This  is  coupled  with  the 
fact  the  virtual  or  constructive  simulation  planner  may  have  to  rely  solely  on  a  computer 
monitor  or  graphic  map  for  inspecting  the  terrain.  In  this  manner,  the  selection  is 
complicated  by  a  lack  of  resources  (time)  and  reduced  information. 


Shortcomings  of  Existing  Planning  Tools 
The  Army  has  also  experienced  dissatisfaction  with  the  current  tools  designed  to 
support  the  use  of  virtual  and  constructive  simulation  training.  Units  from  the  101st 
Airborne  Division  (Air  Assault)  at  Fort  Campbell,  Kentucky  to  the  24th  Infantry 
Division  (Mechanized)  at  Fort  Stewart  Georgia,  have  expressed  dissatisfaction  with  the 
tools  used  to  plan  simulation  based  training  (Barrett,  Telephone  Interview,  1996,  Rossi, 
Telephone  Interview,  1996).  One  of  the  key  shortcomings  identified  by  the  units  in  the 
field  as  well  as  Dr.  Larocque,  the  Director  of  the  National  Simulations  Center  at  Fort 
Leavenworth,  is  the  lack  of  METT-T  responsiveness  within  the  planning  tools. 
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This  dissatisfaction  with  the  planning  tools  occurs  at  the  same  time  the  U.S. 
Army  has  undergone  increased  reliance  on  simulation  based  training  and  the 
establishment  of  extensive  simulation  libraries. 
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Standard  Army  Training  System 


The  Standard  Army  Training  System  (SATS)  is  an  automated  training 
management  tool  the  Army  created  to  ease  the  burden  of  units  in  all  types  of  training. 

The  current  version,  4. 1,  is  being  fielded  to  many  units  now.  SATS-TREDS  (Standard 
Army  Training  System-Training  Exercise  Development  System)  is  a  prototype  training 
exercise  planning  tool  being  developed  for  the  Combined  Arms  Tactical  Trainer  (PM- 
CATT)  at  STRICOM. 

SATS-TREDS  was  designed  to  mimic  the  mental  models  of  a  Subject  Matter 
Expert’s  (SME)  conceptual  planning  development  process  (Stone,  The  Use  of  Computer- 
hasp.d  Planning  to  Enhance  the  Effectiveness  and  Efficiency  of  Simulation-based  Team 
Training  1996).  The  planning  tool  was  developed  using  interviews,  conference  data, 
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rapid  prototyping,  commercial  off  the  shelf  software  and  the  Cognitive  Analysis  Tool 
(Stone,  1996). 

The  system’s  current  capabilities  include  scheduling,  monitoring  unit 
proficiency,  exercise  products,  training  assessment  tools,  a  scenario  library,  and  mission 
and  task  selection.  Though  SATS-TREDS  is  designed  to  be  used  by  a  company, 
battalion  and  brigade  staff,  its  exercise  products  support  all  levels  of  training  from 
brigade  to  platoon. 

Simulation  scenario  libraries,  under  the  current  version  of  SATS-TREDS,  are 
searched  using  tasks  lists.  The  planner  creates  a  task  list  (This  addresses  the  unit’s 
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mission)  by  entering  all  of  the  tasks  he  wishes  to  train.  All  of  the  scenarios  are  then 
compared  to  this  task  list  and  the  scenarios  with  the  highest  number  of  matches  are 
reported  to  the  planner  as  the  most  suitable.  This  gross  search  requires  the  user  to 
manually  examine  each  returned  scenario  for  other  factors  that  will  affect  his  selection 
decision.  For  example,  a  user  must  analyze  each  scenario  in  terms  of  METT-T  factors 
which  include  the  training  unit’s  mission  (Defense  in  sector  vs.  Deliberate  attack),  the 
expected  enemy  (Motorized  rifle  regiment  vs.  Tank  battalion),  the  terrain  in  the  area 
(jungle  vs.  desert),  friendly  troops  that  are  available  (a  pure  infantry  battalion  vs.  and 
armor  task  force)  and  time  available  as  well  as  combat  power  ratios  and  the  training  units 
level  of  proficiency.  This  refinement  process  must  be  analyzed  for  the  possible  formal 
adoption  and  automation. 

As  the  official  scenario  library  is  used,  commanders  will  modify  the  sanctioned 
scenarios  to  make  them  more  useful  to  their  units.  This  process  will  quickly  expand  the 
number  of  scenarios  available  to  the  units  and  make  their  selection  procedure  more 
complex.  It  becomes  more  difficult  to  differentiate  between  similar  scenarios  as  they 
branch  from  a  root  official  scenario.  Therefore,  the  refinement  of  the  scenario  selection 
procedure  by  the  inclusion  of  significant  METT-T  factors  will  improve  the  unit  training 
planning  process.  No  work  has  been  done  on  selecting  training  scenarios  based  on 
METT-T  or  other  factors  instead  of  task  lists.  The  first  step  in  understanding  how 
METT-T  might  impact  on  scenario  selection  begins  with  gathering  information  about  the 
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possible  relationships  between  the  two.  Given  the  diversity  of  U  S.  Army-wide  needs,  a 
methodology  must  be  identified  by  which  to  determine: 

1 .  What  method  of  knowledge  acquisition  is  most  suitable  for  collecting 
information  from  a  widely-  dispersed  non-homogeneous  population? 

2.  What  means  of  analysis  are  the  most  appropriate  for  differentiating 
levels  of  significance  between  the  selection  factors? 

3.  How  can  this  information  be  applied  to  the  selection  of  simulation- 
based  training  scenarios? 

Part  of  this  research  process  will  be  the  quantification  or  classification  of  the  existing 
scenarios  and  user  preferences.  Principles  of  classification  must  be  harnessed  to  ensure 
that  the  classification  system  has  the  granularity  to  support  a  discriminating  search 
process. 

Task  Performance  Support  Codes 

Task  Performance  Support  (TPS)  Codes  were  developed  by  Sherikon  in  support 
of  the  automated  SATS-TREDS  tool  as  part  of  the  Close  Combat  Tactical  Trainer 
program.  Sherikon  developed  this  methodology  to  quantify  the  degree  with  which  a 
simulation  supports  training  a  particular  task  (Sherikon,  Task  Performance  Support 
(TPS')  Codes.  1995).  The  TPS  Code  methodology  adheres  to  the  following  principles: 

1 .  There  must  be  a  single  methodology  which  can  be  applied  to  all 
collective  training  simulations. 
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2.  The  resulting  codes  must  have  sufficient  granularity  to  provide 
reasonable  assurance  of  distinguishing  between  simulation 
environments  at  the  Mission  Training  Plan  level. 

3 .  The  methodology  must  be  efficient,  unambiguous,  repeatable  and 
automated. 

4.  The  methodology  must  be  documented  to  facilitate  reuse  as  new  tasks 
are  added  or  old  ones  are  modified. 

5.  SME  assessments  must  be  retained  in  a  automated  format  for 
traceability  and  consistency  analysis.  (Sherikon,  1995,  pg.  10) 

The  TPS  Code  principles  could  also  be  applied  to  the  problem  of  quantifying  the 
characteristics  of  each  simulation  scenario  in  terms  of  the  scenario’s  association  with  the 
METT-T  factors. 


Methodological  Vision 

An  intuitive  approach  to  quantifying  the  characteristics  of  each  simulation 
scenario  will: 

1.  build  on  successful  previous  simulation  scenario  selection  efforts; 

2.  determine  user  simulation  training  scenario  selection  criteria; 

3.  quantify  selection  criteria  so  that  it  can  be  systematically  applied  to  training 
scenarios  and  the  training  scenario  selection  process  (weights); 
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4.  apply  the  weights  in  a  manner  that  supports  meaningful  discrimination  of 
alternatives; 

5.  ensure  the  process  supports  automation  to  reduce  the  training  planner 
workload. 

While  these  qualities  may  appear  instinctive,  it  also  constitutes  a  vision  whereby 
the  characteristics  of  simulation  scenario’s  are  quantified  to  better  enable  the  user  to 
select  the  optimal  scenario.  The  most  important  parts  of  this  vision  are  the  ability  to 
assign  weights  to  the  selection  factors  and  application  of  the  weights  in  a  manner  that 
supports  discriminating  decision  making. 

The  value  of  determining  the  weights  or  relative  degrees  of  importance  of  the 
simulation  training  scenario  selection  factors  is  that  the  weights  allow  the  selection 
process  to  become  multi  dimensional.  The  tool  can  mimic  the  planners  mental  process  of 
considering  many  different  factors  and  assigning  them  differing  levels  of  importance  to 
the  decision  at  hand. 

The  application  method  is  important  because  it  prepares  a  sound  decision  making 
process  for  automation.  Once  the  process  can  be  automated  it  can  reduce  the  workload 
and  increase  the  accuracy  of  the  user. 


Summary 

Military  training  has  focused  on  live  simulation  training  since  its  inception.  This 
has  lead  to  excellent  tools  and  methods  for  planning  live  training.  Simulation  based 
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training  has  lagged  behind  this  effort  because  of  the  relative  youth  of  the  technologies 
that  have  made  computer  based  training  possible.  Several  categories  of  research  have  to 
be  examined  in  order  to  contribute  to  the  vision  of  simulation  training  planning  and 
scenario  selection.  Areas  of  training,  expert  systems,  METT-T,  information  elicitation, 
statistical  analysis,  and  simulations  have  to  be  examined  next.  These  efforts  will  improve 
the  development  of  the  envisioned  methodology. 
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CHAPTER  3  THE  PREFERENCE-CHARACTERISTIC  CODE 

METHODOLOGY 

Building  on  Previous  Efforts 

Understanding  how  METT-T  and  other  factors  might  effect  selection  of  a 
simulation  training  scenario  begins  with  a  mapping  of  the  steps  taken  by  a  training 
planner.  These  steps  could  be  transformed  into  a  methodology  which  could  be 
subsequently  automated.  This  research  attempts  to  propose  and  demonstrate  the  use  of 
one  such  approach  referred  to  herein  as  the  Preference-Characteristic  Code  (PCC) 
methodology.  The  PCC  methodology  utilizes  SATS-TREDS  and  many  sources. 

The  U.S.  Army’s  current  Training  Management  and  Battle  Focused  Training 
Doctrine  as  well  as  the  continuing  efforts  in  training  development  being  done  by  the 
Army  Research  Institute  (ARI)  support  the  analysis  tools  used  to  plan  live  training.  The 
CCTT  Program  and  other  simulation  based  training  programs  provide  insight  into  the 
future  needs  of  the  commanders.  The  SATS-TREDS  planning  tools  is  the  most  basic 
launching  point  a  new  improved  technique  that  may  be  applicable  to  planning 
simulation-based  training  in  units.. 

SATS-TREDS  provides  the  first  cut  of  simulation  training  scenario  selection. 
PCC  is  designed  to  refine  the  search  conducted  by  SATS-TREDS.  The  PCC 
methodology  may  be  used  as  an  initial  search  as  well. 
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Simulation  Training  Scenario  Selection  Criteria 


Simulation  training  scenario  selection  criteria  needs  to  based  on  U.S.  Army 
doctrine  and  training  practices.  Battlefield  Operating  Systems  (BOS),  Principles  of 
Training,  and  METT-T  are  the  most  promising  expressions  of  U.S.  Doctrine  and  training 
practices  for  selection  criteria  use. 

The  Battlefield  Operating  Systems  are  the  major  functions  a  commander  uses  to 
ensure  he  accomplishes  his  mission  (Training  Management,  1991).  Intelligence, 
Maneuver,  Fire  Power,  Mobility/Counter  Mobility/Survivability,  Air  Defense,  Combat 
Service  Support,  and  Command  and  Control  are  all  of  the  systems  present  on  the 
battlefield.  These  systems  may  not  be  appropriate  or  germane  to  all  training  situations 
because  of  unit  “pure”  training  that  may  be  done  in  simulations  such  as  the  Mobile 
Target  Simulator  (MTS)  for  Air  Defense  units  or  the  Advanced  Target  Gunnery  System 
(AGTS)  for  Armor..  The  fact  that  they  cannot  be  consistently  applied  to  all  cases  could 
lead  to  a  loss  of  granularity  in  the  selection  process. 

The  nine  Principles  of  Training  act  as  imperatives  dictating  the  conduct  of 
training  (Training  Management,  1991).  The  concepts  of  training  as  combined  arms, 
training  as  you  fight,  training  to  maintain,  training  using  multi-echelon  techniques  are 
easy  to  quantify  and  identify.  However  using  appropriate  doctrine,  training  to  challenge, 
and  using  performance-oriented  training  would  be  hard  to  quantify.  The  inability  to 
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apply  all  of  the  criteria  to  reasonable  degree  to  all  of  the  scenarios  and  preferences 
eliminated  this  as  an  option. 

Not  only  was  METT-T  the  recommended  medium  from  the  units  in  the  field,  it 
seems  that  the  elements  of  METT-T  are  most  applicable  to  the  problem  at  hand. 

Mission,  Enemy,  Terrain,  Troops,  and  Time  can  be  defined,  quantified  and  applied  to  the 
scenarios  and  planner  preferences.  METT-T  is  applied  to  course  of  action  analysis  and 
other  tactical  decisions.  Though  METT-T  can  be  broadly  applied  both  to  the  scenario’s 
and  the  users  preferences,  it  should  be  bolstered  with  considerations  affecting  unit 
assessment,  previous  use  of  a  scenario,  difficulty,  and  available  simulators. 

METT-T  Application 

The  five  METT-T  factors  can  easily  be  broken  down  into  more  precise  sub¬ 
categories  that  are  easily  recognizable  to  military  trainers.  Table  3  breaks  the  five  factors 
into  sub -categories  and  adds  some  general  factors  of  interest. 


TABLE  3  METT-T  CATEGORIES  AND  GENERAL  SELECTION  FACTORS 


METT-T  Category 

Selection  Factor 

Mission 

Task 

Combat  Power  Ratio 

Enemy 

Enemy  Composition 

Enemy  Task  Organization 
Enemy  Equipment 

Enemy  Training  Level 

Enemy  Mission 

Time 

Exercise  Preparation  Time 
Mission  Planning  Time 

Terrain 

Terrain 

Weather 

Light  Data 

Troops 

Friendly  Composition 

Friendly  Equipment 

Adjacent  Unit 

Supported  Observation 

Devices 

General 

Previous  Use  of  a  Scenario 
Difficulty 

Simulator 

Assessment 

The  selection  factors  may  be  broken  down  into  meaningful  divisions.  These 
divisions  allow  the  scenarios  and  the  users  preferences  to  be  coded.  Coding  supports 
automation  of  the  selection  processes. 


Quantification  of  Selection  Criteria  and  The  Manner  of  Application 


The  selection  factors  from  Table  4  are  broken  down  into  categories  based  on  the 
U.S.  Army’s  training  doctrine  and  methods.  These  selection  factors  can  be  used  to 
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describe  user  preferences  sought  in  a  scenario  that  maybe  used  for  basis  of  training. 
Similarly,  the  available  simulation  training  scenarios  might  be  evaluated  and  classified  to 
indicate  whether  or  not  they  offer  the  characteristics  sought  by  the  trainer. 

The  list  of  user  preferences  may  be  grouped  to  form  a  matrix  of  preferences 
referred  to  as  the  User  Preference  matrix  (UPM)  or  the  User  Preference  Code  (UPC). 

The  list  of  scenario  characteristics  can  be  grouped  to  form  the  Scenario  Characteristic 
matrix  (SCM)  or  Scenario  Characteristic  Code  (SCC).  Given  the  importance  of  the 
factors  in  Table  4  both  the  UPC  and  SCC  structured  around  these  factors  as  discussed 
below. 


Mission 

Task: 

The  U.S.  Army  breaks  all  operations  down  into  a  series  of  collective  and 
individual  tasks  (Training  Management,  1991)  This  type  of  decomposition  supports  the 
analysis  of  large  complex  missions  in  terms  of  the  functions  or  tasks  the  subordinate 
elements  must  accomplish. 

This  type  of  crosswalk  relationship  carries  over  into  the  Army  Training 
Evaluation  Plan  (ARTEP)  Mission  Training  Plan  (MTP)  manuals.  MTP  manuals  contain 
the  standards  of  performance  of  these  tasks  along  with  the  breakdown  of  who  or  which 
element  must  perform  each  sub-task. 

When  considering  how  military  units  might  go  about  applying  simulator  training 
systems  to  training  a  unit  in  a  particular  task,  the  planner  first  begins  with  identifying  the 
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tasks  to  be  trained.  This  task  list  is  the  basis  of  the  SATS-TREDS  selection  process.  For 
the  purpose  of  this  research  one  task  can  be  determined  the  most  important  task  to  be 
trained.  This  selection  can  be  codified  in  the  Task  matrix,  [tg].  The  Task  matrix  is  later 
intersected  with  a  Scenario  Task  matrix  [Ty]  which  contains  all  of  the  tasks  supported  by 
the  training  scenario.  If  the  intersection  of  the  matrixes  results  in  the  null  set,  the 
scenario  in  question  does  not  support  training  the  task  to  be  trained. 

This  relationship  between  the  tasks  to  be  trained  and  the  tasks  supported  by  the 
scenario  yields  the  first  element  of  the  User  Preference  Code  (UPC).  If  the  scenario  does 
not  support  the  required  task,  U1;1=  1.  Otherwise,  U1;1  =  0. 

If  there  is  no  particular  task  that  must  be  trained  Un=  0. 

Combat  Power  Ratio: 

Combat  Power  Ratio  (CPR)  refers  to  the  aggregated  strength  levels  of  the 
opposing  forces  in  the  simulation  training  scenario  expressed  as  a  ratio  of  friendly  forces 
to  enemy  forces.  There  exists  normally  acceptable  ratios  for  certain  types  of  operations. 
An  example  of  this  is  that  U.S.  Army  forces  prefer  to  attack  if  the  CPR  is  equal  to  or 
greater  than  3:1.  Ratios  above  this  (3:1)  favor  the  attacker,  while  ratios  below  this  favor 
the  defender. 

The  user’s  selection  of  CPR  above  or  below  the  doctrinal  level  determines  the 
value  of  the  UPC  element  U1>2..  Similarly  the  ratio  that  exists  in  the  scenario  determines 
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the  Scenario  Characteristic  Code  (SCC)  element  S1>2 .  The  actual  values  for  the  element 
are  found  in  Table  9  in  Appendix  B. 


Enemy 

Enemy  Composition: 

It  is  impossible  to  create  a  finite  set  that  covers  all  aspects  of  enemy  composition. 
To  streamline  this  process  the  U.S.  Army  has  developed  various  templates  that  are  based 
on  the  enemy’s  characteristics.  Though  the  Soviet  Union  no  longer  exists,  its  previous 
activities  within  its  sphere  of  influence  affected  the  equipment  and  doctrinal  structure  of 
many  potential  adversaries  throughout  the  world  (Soviet  Army  Equipment,  Organization, 
and  Operations,  1991).  For  this  reason  its  former  configurations  and  tactical  dispositions 
make  reliable  starting  -points  for  enemy  classifications. 

The  User’s  Preference  Code  (Ut  3)  for  enemy  composition  as  well  as  the  Scenario 
Characteristic  Code  (S13)  are  based  on  theses  models.  The  coded  values  used  for  these 
elements  are  drawn  from  Table  10  in  Appendix  B.  In  the  table  the  expected  major 
enemy  organizations  are  listed  along  with  a  coded  value  to  represent  them. 

Enemy  Task  Organization: 

The  enemy’s  task  organization  is  dependent  to  a  large  degree  upon  the  role  it 
plays  within  the  enemy’s  larger  plan.  It  is  convenient  to  look  at  this  aspect  in  terms  of 
where  the  enemy  unit  exists  in  the  enemy’s  maneuver  structure  (Soviet  Army  Equipment, 
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Organization,  and  Operations,  1991).  For  the  purpose  of  templating  the  enemy  this 
method  is  very  useful. 

The  UPC  (Ulj4)  and  the  SCC  (S14)  elements  are  determined  from  Table  11  in 
Appendix  B.  In  the  table  the  possible  elements  of  the  enemy  organizations  are  listed 
along  with  a  coded  value  to  represent  them. 


Enemy  Equipment: 

The  enemy’s  equipment  can  be  generally  categorized  in  terms  of  their  most  lethal 
maneuver  systems.  The  most  lethal  systems  available  to  most  of  our  potential  enemies 
are  the  T-80  main  battle  tank,  the  BMP-3  infantry  fighting  vehicle  and  the  Holkum  attack 
helicopter  (Soviet  Army  Equipment,  Organization,  and  Operations,  1991).  More 
common,  and  less  capable,  systems  in  these  categories  are  the  T-72  series  tanks,  the 
BMP-2  infantry  fighting  vehicle,  and  the  Hind  series  attack  helicopter.  If  the  enemy  force 
has  certain  combinations  of  these  systems,  assumptions  can  be  made  about  the  other 
indirect  fire  and  combat  systems  he  possesses. 

Knowing  that  the  enemy  posses  certain  weapon  systems  indicates  some  of  their 
likely  courses  of  actions,  capabilities,  and  tactics. 

The  UPC  (U1>5)  and  the  SCC  (S1;5 )  elements  are  determined  from  Table  12  in 
Appendix  B.  In  the  table  the  major  weapon  system  classifications  are  listed  along  with  a 
coded  value  to  represent  them. 
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Enemy  Training  Level: 


Most  training  simulations  use  stochastic  methods  to  determine  target  hits,  kills 
and  detection  (Combat  Simulation  Laboratory,  undated).  Increasing  or  decreasing  these 
levels  may  effect  the  effectiveness  of  the  enemy  elements  which  could  be  taken  as 
improved  or  depressed  levels  of  training.  These  characteristics  can  be  important  when  a 
military  planner  prepares  for  battle  or  training. 

The  UPC  (Ui  6)  and  the  SCC  (S16)  elements  are  determined  from  Table  13  in 
Appendix  B.  In  the  table  the  levels  of  probabilities  of  hit,  kill  and  detection  are  listed 
along  with  a  coded  value  to  represent  them. 


Enemy  Mission: 

The  enemy’s  mission  has  a  significant  impact  on  the  activities  he  may  engage  in 
during  a  training  session.  The  former  Soviet  Union  and  their  client  states  only  executed 
six  (6)  types  of  missions  (Soviet  Army  Equipment,  Organization,  and  Operations,  1991). 

The  UPC  (U1>7)  and  the  SCC  (S17)  elements  are  determined  from  Table  14  in 
Appendix  B.  In  the  table  the  possible  enemy  missions  are  listed  along  with  a  coded  value 
to  represent  them. 
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Time 


Exercise  Preparation'Time: 

The  amount  of  time  and  effort  involved  in  the  preparation  of  a  training  exercise 
can  effect  its  selection.  If  an  organization  has  a  small  amount  of  time  to  prepare  for 
training  (a  National  Guard  or  Reserve  Unit)  they  may  prefer  to  select  simulation  training 
scenarios  that  allow  minimal  unit  planning  prior  to  training  execution.  Scenarios  with 
minimal  preparation  time  may  include  very  detailed  graphics,  training  schedules, 
coordination  check  lists. 

The  UPC  (U1>8)  and  the  SCC  (Slg)  elements  are  determined  from  Table  15  in 
Appendix  B.  In  the  table  the  levels  of  exercise  preparation  time  are  listed  along  with 
coded  values  to  represent  them. 

Mission  Planning  Time: 

Mission  planning  time  refers  to  the  amount  of  tactical  planning  and  operations 
order  development  the  unit  must  do.  Units  that  are  incomplete  staffs  or  have  the  staff 
dedicated  to  other  missions  may  wish  to  execute  supplied  orders.  Similarly,  the  units 
staff  may  not  be  prepared  to  execute  the  mission  planning  process  in  the  available  time 

The  UPC  (U19)  and  the  SCC  (S19 )  elements  are  determined  from  Table  16  in 
Appendix  B.  In  the  table  the  levels  of  mission  planning  required  are  listed  along  with 
coded  values  to  represent  them. 
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Terrain 


Terrain: 

The  United  State  Army  must  be  prepared  to  execute  its  missions  anywhere  in  the 
world.  This  drives  the  requirement  that  the  U.S.  Army  train  on  all  different  types  of 
terrain,  weather,  and  light  conditions.  For  this  reason  the  planner  needs  the  ability  to 
distinguish  between  the  various  land  forms  the  unit  could  train  on. 

The  UPC  (Uj  10)  and  the  SCC  (SI;  i0)  elements  are  determined  from  Table  17  in 
Appendix  B.  In  the  table  the  major  land  forms  are  listed  along  with  coded  values  to 
represent  them. 


Weather: 

Weather  affects  numerous  aspects  of  tactical  operations.  Trafficability, 
equipment  performance,  tactics  and  procedures  are  effected  by  weather.  Training 
planners  may  want  to  gauge  their  units  ability  to  adapt  to  these  different  conditions. 

The  UPC  (UU1)  and  the  SCC  (S1U  )  elements  are  determined  from  Table  18  in 
Appendix  B.  In  the  table  the  weather  conditions  are  listed  along  with  coded  values  to 
represent  them. 
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Light  Data: 


Light  data  affects  numerous  aspects  of  tactical  operations.  Equipment  performance, 
tactics  and  procedures  are  effected  by  light  data.  Training  planners  may  want  to  gauge 
their  units  ability  to  adapt  to  these  different  conditions. 

The  UPC  (UU2)  and  the  SCC  (SU2)  elements  are  determined  from  Table  19  in 
Appendix  B.  In  the  table  the  light  conditions  are  listed  along  with  coded  values  to 
represent  them. 


Troops 

Friendly  Composition: 

Though  the  system  will  take  the  unit  data  that  is  in  use  as  a  default,  the  Friendly 
Unit  Composition  can  be  altered.  This  allows  the  commander  to  plan  training  for  the 
units  one  level  down.  This  could  happen  in  preparation  for  an  evaluated  training  exercise 
or  assigning  directed  training. 

The  UPC  (Uu3)  and  the  SCC  (S,  13)  elements  are  determined  from  Table  20  in 
Appendix  B.  In  the  table  the  units  are  listed  along  with  coded  values  to  represent  them 
(Organization  of  the  Army  in  the  Field,  1991). 
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Friendly  Equipment: 


1 

I 


The  system  default  would  be  set  for  the  unit’s  Table  of  Organization  and 
Equipment  (TO&E).  However,  units  periodically  deploy  and  unitize  pre-positioned 
equipment.  The  propositioned  equipment  may  be  different  from  the  equipment  that  the 
unit  has  at  its  home  station.  The  ability  to  choose  different  equipment  could  affect  the 
scenario  selection. 

The  UPC  (UU4)  and  the  SCC  (S!  14)  elements  are  determined  from  Table  21  in 
Appendix  B.  In  the  table  friendly  equipment  categorize  are  listed  along  with  coded 
values  to  represent  them. 

Adjacent  Unit: 

The  U.S.  Army  has  become  increasingly  involved  in  multi-national  operations 
such  as  Somalia,  Haiti,  Macedonian,  and  Bosnia.  When  training  for  such  missions  it  may 
become  important  to  be  knowledgeable  about  the  allies’  equipment,  tactics  ,  and 
capabilities. 

The  UPC  (UU5)  and  the  SCC  (S115)  elements  are  determined  from  Table  22  in 
Appendix  B.  In  the  table  possible  allies  are  listed  along  with  coded  values  to  represent 
them. 


39 


Supported  Observation  Devices: 


All  simulations  do  not  have  the  same  capabilities  to  support  or  model  different 
battlefield  sensors  (Combat  Simulation  Laboratory,  undated).  This  demands  that  the 
planner  have  the  ability  to  select  simulations  based  on  their  capabilities. 

The  UPC  (Uu6)  and  the  SCC  (Sl  l6)  elements  are  determined  from  Table  23  in 
Appendix  B.  In  the  table  sensors  are  listed  along  with  coded  values  to  represent  them. 

General  Category 

Previous  Use  of  a  Scenario: 

The  value  of  training  can  be  degraded  if  the  training  audience  constantly  uses  the 
same  situations  to  learn  (Anderson,  1993).  However,  repeating  an  exercise  that  was 
incorrectly  performed  can  boost  learning.  Depending  on  the  number  of  attempts  to 
complete  a  single  scenario  the  planner  may  wish  to  exclude  or  repeat  a  scenario. 

The  user’s  desired  Previous  Use  of  a  Scenario  determines  the  relationship 
between  the  Previous  Use  matrix,  [Py],  and  the  scenario  number.  If  the  user  wants  to 
reuse  a  scenario,  the  Preference-Characteristic  Technique  (PCT)  examines  the 
intersection  of  the  set  of  used  scenarios  and  the  current  scenario.  If  the  intersection  is 
the  null  set,  UU7  =  1,  otherwise  U117=0. 

If  the  user  does  not  want  to  reuse  a  scenario  and  the  intersection  is  the  null 
set  then  U117=  0,  otherwise  UU7=  1. 
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Level  of  Difficulty: 


In  an  effort  to  stress  the  training  unit  or  to  utilize  a  Crawl-Walk-Run  training 
methodology  it  is  needed  to  know  the  degree  of  difficulty  associated  with  a  simulation 
training  scenario.  Though  the  issue  of  how  to  objectively  quantify  the  degree  of 
difficulty  is  unresolved,  there  is  a  benefit  to  be  gained  throughout  the  application  of  the 
information. 

The  UPC  (U1>18)  and  the  SCC  (Sx  18)  elements  are  determined  from  Table  24  in 
Appendix  B.  In  the  table  levels  of  difficulty  are  listed  along  with  coded  values  to 
represent  them. 

Simulator: 

There  may  be  constraints  on  the  training  plan  that  limit  the  user  to  a  specific  type 
of  training  simulator.  This  could  be  a  resource  constraint  or  an  issue  of  the  capabilities 
of  a  simulator  to  support  a  particular  exercise.  This  generates  the  need  for  this  element. 

The  UPC  (Uu9)  and  the  SCC  (Slsi9)  elements  are  determined  from  Table  25  in 
Appendix  B.  In  the  table  simulator  systems  are  listed  along  with  coded  values  to 
represent  them. 


Assessment: 

U.S.  Army  training  strategies  focus  on  sustaining  proficiency  in  tasks  that  are 
executed  well  and  improving  performance  in  tasks  that  are  not  executed  well.  Since  all 
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training  is  evaluated  and  unit  performance  on  critical  tasks  is  formally  evaluated 
quarterly,  it  is  possible/desirable  to  give  preference  to  simulation  training  scenarios  that 
train  tasks  that  have  been  identified  during  assessment  as  not  being  well  executed 
(Training  Management,  1991). 

The  user’s  desired  use  of  the  unit  assessment  data  produces  the  following 

values  of  U120: 

If  the  user  wants  to  consider  Unit  Assessment  data:  The  tool  examines  the 
intersection  of  the  set  of  the  Scenario  Task  Matrix,  [Ty],  and  the  Untrained 
Matrix,  [Fy].  The  Untrained  Task  matrix  is  a  set  of  tasks  that  the  unit  has 
identified  through  the  training  management  process  that  they  do  not  execute  well. 
If  the  intersection  is  the  null  set,  U120  =  1,  otherwise  U1;20  =  0. 

If  the  user  does  not  want  to  consider  unit  assessment  data ,  then  Uj  2o =  0. 

The  elements  of  the  Scenario  Characteristic  Code  matrix,  [Sy], 
correspond  to  those  of  the  User  Preference  Code  matrix,  [Uy].  The  elements  that  relate 
directly  to  the  past  actions  of  the  training  unit  in  the  UPC  matrix,  Ulfl,  Uu7,  U1>20,  all 
equal  zero  (0)  in  the  SCC  matrix,  S2>1,  Sji17,  S1;20. 
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Preference-Characteristic  Matrix  Technique 


The  Scenario  Characteristic  Code  (SCC)  then  is  a  1  x  20  matrix,  [Sy],  whose 
elements,  Sy,  are  coded  qualities  of  the  scenario  in  terms  of  the  METT-T  factors  of 
mission,  enemy,  time,  terrain,  and  troops. 

The  SCC  elements  are  first  organized  by  grouping  all  of  the  factors  according  to 
their  METT-T  associations  as  shown  in  Table  3. 

The  scenarios  are  then  analyzed  according  to  these  factors  by  the  standards  and 
categories  established  in  Tables  9  through  25.  The  elements  characteristics  are  compared 
to  the  properties  on  the  tables  and  categorized.  The  various  categories  on  the  charts  are 
designed  to  be  exhaustive.  If  in  evaluating  a  scenario  or  preference  it  is  not  possible  to 
place  it  into  a  category  then  an  additional  category  must  be  added. 

Each  selection  factor  is  discussed  above  along  with  an  accompanying  table  (Table 
9  through  25)  that  lists  the  possible  values  that  enter  the  matrices  based  on  the  scenario’s 
characteristics  or  user’s  preferences. 

The  SCC  is  subtracted  from  the  User  Preference  Code  (UPC),  which  is  also  a  1 
x  20  matrix,  [Uy].  The  elements  of  the  [Sy]  are  determined  by  examining  the  scenario, 
while  the  [Uy]  is  determined  by  the  user. 

The  Scenario  Task  matrix,  [Ty],  is  a  listing  of  all  of  the  training  tasks  associated 
with  a  scenario. 

The  Task  matrix,  [ty],  is  a  matrix  containing  the  most  important  training  task  as 
determined  by  the  user. 
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The  Previous  Use  matrix,  [Py],  consists  of  a  list  of  all  of  the  scenarios  used  by  the 

unit. 

The  Untrained  Matrix,  [Fy],  consists  of  a  list  of  all  of  the  METL  tasks  that  the 
unit  has  incorrectly  performed  or  for  which  the  unit  has  failed  to  perform  one  or  more 
critical  subtasks.  This  classification  of  Untrained  follows  Army  guidance  (Combined 
Arms  and  Service  Staff  School,  1991). 

The  Calculation  of  the  Weighted  Result 

Once  the  UPC  and  SCC  matrices  are  constructed,  the  SCC  matrix  is  subtracted 
from  the  UPC  matrix.  The  resulting  Difference  matrix,  [Dg],  is  examined  for  the  value 
of  the  elements,  Dg  . 

If  Dy  is  equal  to  0,  then  Ug  =  1 . 

If  Dy  is  equal  to  (-1)  Sy,  then  Ug  =  1. 

Otherwise,  Ug  =  0. 

The  assignment  of  the  value  1  to  Ug  if  Dy  is  equal  to  0  or  if  Dy  is  equal  to  (-1) 

S: :  is  made  so  that  a  scenario  receives  a  benefit  if  the  matrix  element  of  UPC  is  the  same 
as  the  element  of  the  SCC  or  if  the  user  expressed  no  preference  in  that  category.  For 
example,  if  the  user  wants  a  scenario  that  has  U.S.  Marines  as  the  adjacent  unit  and  the 
user  has  no  preference  for  supported  imaging  devices,  the  UPC  would  have  a  value  of  9 
for  [U  1;  15  ]  and  a  value  of  0  for  [U  1;  16  ].  If  one  of  the  compared  simulation  training 
scenarios  had  values  for  those  elements  of  [U  i  15  ]  =  10  and  [U  i;  i6  ]  =  2  (-2  =  -Sy)  the 


44 


values  for  the  Difference  matrix  would  be  [D  115  ]  =  0  and  [D  116]  =  -2.  The  adjacent 
unit  characteristic  should  lead  to  positive  consideration  because  it  is  the  desired 
characteristic  and  the  observation  devices  should  have  a  positive  impact  because  the  user 
had  no  preference.  The  scenario  receives  no  benefit  if  the  scenario  characteristic  is 
different  from  the  user’s  expressed  preference. 

The  revised  UPC  matrix  could  be  multiplied  by  a  set  of  weights,  the  Weight 
matrix  [Wj;1].  The  Weight  matrix  accounts  for  the  differing  levels  of  importance  the 
selection  factors  have. 


W2.1 

r  2°  1 

This  results  in  Rank  matrix  [R,  J.  [/(  [«„,»,  2 . ux 20 

3«,1_ 

_k= 1 

(equation  1) 

The  Rank  matrix  is  a  single  number  whose  value  is  proportional  to  the  scenario’s 
agreement  with  the  user’s  preferences.  A  Rank  matrix  can  be  calculated  for  each 
scenario  that  SATS-TREDS  has  returned  as  matching  a  number  of  tasks.  The  Rank 
matrix  now  permits  the  automated  differentiation  of  simulation  training  scenarios  based 
on  user  METT-T  preferences.  An  example  illustrating  the  use  of  the  Rank  matrix 
appears  in  Chapter  6-Data  Analysis. 
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The  success  of  the  methodology  hinges  on  the  ability  to  define  a  discriminating 
Weight  matrix.  The  Weight  matrix  must  be  developed  through  the  elicitation  of 
knowledge  from  the  expected  using  population  or  a  sample  of  that  population.  The 
Weight  matrix  will  allow  the  appropriate  level  of  significance  to  be  applied  to  each 
dimension  (selection  factor). 


Summary 

It  is  possible  to  define  a  series  of  matrices  that  describe  the  tool  user  and  his 
desires,  training  audience,  the  units  training  activities  and  the  available  simulation 
training  scenarios.  These  matrices  can  then  be  manipulated  along  with  an  objective 
Weight  matrix  to  determine  the  best  simulation  training  scenario  to  meet  the  users 
preferences.  The  key  to  the  applicability  of  the  matrix  technique  is  the  evaluation  of  the 
Weights  matrix. 


46 


CHAPTER  4  A  TECHNIQUE  FOR  EVALUATING  THE  WEIGHTS  MATRIX 


This  research  effort  produces  weights  for  METT-T  factors.  Those  weights  form 
the  basis  through  which  Scenario  Characteristic  Codes  (SCC’s)  can  be  used  by  an 
automated  tool  to  distinguish  between  simulation  training  scenarios  that  support  equal 
numbers  of  training  tasks.  This  is  different  from  SATS-TREDS  which  currently  sorts 
solely  on  the  number  of  tasks  supported  by  the  scenario. 


Theoretical  Approach  to  Gathering  Information  Needed  to  Refine  Scenario  Selection 

It  would  be  possible  to  gather  the  information  needed  to  augment  the  current 
scenario  selection  system  through  many  means.  The  information  concerning  the  use  of 
METT-T  factors  in  selecting  a  training  scenario  could  be  acquired  using  a  subject  matter 
expert,  a  Delphi  group  technique,  or  a  survey.  After  the  information  is  gathered  a 
development  process  has  to  be  selected  in  order  to  produce  an  exercise  development  tool. 
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The  Use  of  Subject  Matter  Experts 


The  use  of  a  subject  matter  expert  (SME),  with  a  great  deal  of  experience  with 
training  in  units,  would  yield  valuable  information  in  a  very  timely  manner.  A  single 
person’s  input  could  be  easily  quantified  and  developed  into  a  useful  tool.  A  useful  three 
phase  method  for  extracting  this  information  is  commonly  used;  unstructured  interviews, 
recording  of  the  task,  and  structured  interviews  (Williams,  1995).  The  first  phase  is  to 
conduct  unstructured  interviews  focusing  on  the  general  task  being  analyzed.  This  initial 
interview  is  limited  by  the  extent  of  domain  knowledge  held  by  the  knowledge  engineer 
(Kotnour,  Design.  Development,  and  Testing  of  an  Automated  Knowledge  Acquisition 
Tool  to  Aid  Problem  Solving.  Decision  Making,  and  Planning.  1992).  This  is  followed 
by  recording  (using  video  tape  or  other  devices)  the  actions  of  the  expert  while  engaged 
in  the  task.  The  recording  is  then  analyzed  by  the  SME  along  with  the  researcher.  This 
analysis  consists  of  structured  interviews  in  which  the  SME  describes  the  process 
portrayed  in  the  recording  (Welbank,  An  Overview  of  Knowledge  Acquisition  Methods, 
1990). 

Though  great  insight  can  be  gained  through  this  technique,  a  single  SME’s 
opinion  may  not  prove  to  be  representative  of  the  entire  population.  The  application  of 
the  information  is  going  to  vary  over  the  entire  Army  at  a  tactical  level  of  application. 
Though  a  person  could  easily  become  an  expert  in  a  single  aspect  of  the  battlefield  and 
training,  at  this  level  the  unique  requirements  of  the  different  military  specialties  and 
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their  associated  idiosyncrasies  would  not  support  the  formulation  of  a  broadly  utilized 
planning  tool. 


Group  Techniques 

The  Delphi  or  Nominal  Group  techniques  may  also  suffer  from  a  lack  of  a  wide 
base  of  knowledge.  The  Nominal  Group  Technique  (NGT)  utilizes  a  group  of  five  to 
nine  experts  in  a  structured  environment.  The  participants  are  prompted  to  compose  a 
list  of  issues  concerning  a  given  problem.  After  each  member  of  the  group  shares  the 
factors  they  listed,  the  group  discusses  each  item  to  produce  a  common  definition.  The 
experts  then  rank  order  the  choices,  and  the  facilitator  tabulates  the  data  and  produces  a 
ranked  list.  After  group  discussion  of  the  preliminary  ranked  list  a  final  vote  is  taken  and 
a  final  rank  order  is  produced  (Delbecq,  1975). 

Like  the  NGT,  the  Delphi  process  is  used  for  aggregating  group  judgment  and  for 
distilling  information  on  highly  complex  problems  (Adler  and  Ziglio,  1996).  The  Delphi 
technique  differs  from  the  NGT  in  that  the  participants  do  not  interact  or  discuss  the 
issues  that  are  presented.  The  facilitator  simply  gives  each  participant  an  issue  statement 
in  the  form  of  a  survey.  The  survey  is  then  collected  and  the  facilitator  lists  the  ideas  of 
the  experts  and  returns  this  new  list  to  the  individuals  for  ranking.  A  ranked  list  is  then 
produced  as  in  the  NGT  (Delbecq,  1975).  The  lack  of  discussion  of  the  issue  between 
the  experts  makes  this  technique  less  desirable.  Strengths  of  the  Delphi  method  are  that 
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it  can  be  used  in  settings  where  the  participants  cannot  be  brought  together,  it  is 
impossible  for  a  forceful  advocate  of  a  view  to  dominate  discussion,  and  its  structure 
reduces  the  possibility  of  “group  think”  (Adler  and  Ziglio,  1996). 

Though  the  collection  of  experts  would  broaden  the  applicability  of  the  resulting 
tool,  the  results  are  still  constrained  to  a  smaller  population  of  the  whole.  The  values 
generated  by  the  exercise  would  also  be  “averaged”  over  the  whole  population.  This 
means  that  the  method  would  be  less  able  to  determine  the  needs  of  smaller  segments  of 
the  total  population. 

The  Delphi  and  Nominal  Group  Techniques  are  also  time  and  resources  intensive. 
The  experts  must  be  committed  to  the  solution  of  the  problem,  willing  to  dedicate  large 
blocks  of  time,  and  must  be  brought  together  at  one  location.  This  manipulation  of  the 
participants  drives  up  the  cost  of  the  process. 


Surveys 

The  use  of  a  survey  would  allow  for  the  elicitation  of  information  from  a  much 
larger  and  more  diverse  population.  This  much  larger  population  would  support  further 
analysis  into  sub  groups  characteristics.  The  cost  associated  with  the  data  collection 
would  also  be  much  less  than  the  other  methods.  However,  the  collection  technique 
would  suffer  in  timeliness. 
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Survey  and  Question  Design 


Three  major  requirements  of  a  survey  is  that  it  is  unambiguous,  easily 
understood,  and  returned.  There  are  many  methods  that  can  be  used  for  implementing 
the  survey  depending  on  its  use  and  the  resources  associated  with  the  project.  The  main 
types  of  surveys  are  random  sampling  surveys  and  directed  surveys  (Lehtonen  and 
Pahkinen,  Practical  Methods  for  Design  and  Analysis  of  Complex  Surveys.  1995).  These 
are  implemented  using  mailed  surveys,  telephone  interviews,  and  personal  interviews. 
There  are  advantages  associated  with  each  type  of  survey  and  the  associated 
implementation  methods. 

Random  sampling  surveys  are  used  for  definitive  results  over  an  entire  population 
or  survey  frame.  They  are  expensive  in  administering  because  of  the  low  response  rate 
and  high  number  of  returns  needed  to  establish  validity.  Directed  surveys  are  used  in 
pilot  studies  (Fink,  1995). 
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Table  4  compares  the  various  survey  implementation  strategies. 

TABLE  4  SURVEY  TECHNIQUE  COMPARISONS 

Aspect  of  Survey  Mailed  Telephone  Face  to  face  WebPage 

Survey  interview  interview _ Surveys 

Administrative _ 

Cost  Low  Low/Medium  High _ Medium/High 

Length  of  data  Long  Short  Medium/Long  Long 

collection  period _ 

Geographic  Wide  May  be  wide  Clustered  Very  Wide 

distribution 


Length  of 
survey 

Short 

(4-12  pages) 

Medium/Long 
1/4-3/4  hour 

Long 

1/2-1  hour 

Short 

(4-12  pages) 

Complexity  of 
questions 

Simple  to 
moderate 

May  be 
complex 

May  be 
complex 

Simple  to 
moderate 

Sensitive  topics 

Good 

Fair/good 

Fair 

Good 

Non  threatening 
questions 

Good 

Good 

Good 

Good 

Data  Quality 


Issues _ _ 

Sampling  frame  Low  Low  Low  Low 

bias _ 

Response  rate  45%-65%  60%-90%  65%-95% _ 45%-65% 

Response  bias  Medium*  Low _ Low _ Medium* 

Control  of  Poor  Fair  Good  Poor 

response  situation _ 

Quality  of  Fair/good  Very  good  Very  good  Fair/good 

recorded 

response _ | _ | _ 

*Czaia  indicates  that  response  bias  is  lower  in  more  educated  respondents.  (Czaja, 

1996) 
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It  takes  approximately  400  responses  from  the  population  to  achieve  a  0.05  level 
of  confidence  that  the  results  of  the  survey  accurately  portray  the  beliefs  of  the 
population  ICzaia.  Designing  Surveys,  1996). 

The  number  of  valid  responses  needed  to  determine  a  valid  output  from  a  survey 
is  determined  by  the  size  of  the  total  population,  the  probability  level,  the  expected 
variance,  and  the  confidence  interval  that  is  acceptable  within  the  study. 


t2(p*q)  _  Probability  level  *  variance 
d2  confidence  interval 


distribution 


Where: 


(equation  2) 


n  =  The  sample  size  or  the  number  of  completed 
interviews  with  eligible  elements. 

t2  =  The  squared  value  of  the  standard  deviation  score 
that  refers  to  the  area  under  a  normal 
of  values. 

p  =  The  percentage  category  for  which  we  are  computing 
the  sample  size. 

q=  1-p. 

d  =  The  value  of  one  half  of  the  precision 

interval  around  the  sample  estimate. 


This  formula  is  applicable  for  surveys  of  less  than  5%  of  the  total  population.  If 
more  than  5%  of  the  total  population  is  surveyed,  then  a  finite  population  factor  is 
applied  (Cochran,  Sampling  Techniques,  1977). 
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V  N)  d 2 

(equation  3) 

Where : 

N  =  The  size  of  the  eligible  population 

All  other  variables  are  the  same  as  in  equation  1 . 

Calculating  the  required  sample  size  for  a  random  sample  of  the  eligible 
population  is  made  using  the  first  equation  without  the  finite  population  factor. 

Assuming  a  desired  95%  probability  level,  a  0.05  confidence  interval,  and  a  percentage 
category  for  which  we  are  computing  a  sample  size  of  50%  yields  a  sample  size  of  384. 
The  initial  percentage  category  is  assumed  to  be  50%  because  there  is  a  lack  of 
information  about  the  population.  The  calculations  following  the  pilot  survey  will  update 
this  value.  At  an  expected  response  rate  of  45%  the  number  of  unsolicited  mailings 
becomes  quite  large  (854). 

The  directed  survey  is  much  less  costly  to  use  and  results  in  a  much  higher 
response  rate.  The  higher  response  rate  eliminates  some  biases  but  the  method  is  best 
used  when  doing  pilot  surveys  (Fink,  1995). 

Ronald  Czaja  and  Johnny  Blair  (1996)  indicate  that  the  survey  should  not  exceed 
10  pages  in  length,  should  appear  uncluttered,  should  provide  easily  identifiable  answer 
spaces.  They  continue  to  establish  that  the  survey  should  have  an  introduction  that 
explains  the  purpose  of  the  survey  and  should  engender  the  subject  to  complete  the 
survey.  A  respondent  section  that  allows  the  information  to  be  collected  on  the  subject  of 
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the  survey.  This  is  the  area  where  demographic  information  (independent  variables)  is 
gathered.  The  substantive  questions  follow.  Theses  questions  are  designed  to  measure 
the  beliefs  of  the  subjects  (dependent  variables)  (Fink,  The  Survey  Handbook.  1995). 
The  survey  was  constructed  so  as  to  have  construct  validity,  so  as  to  establish  that  people 
with  different  characteristics  have  different  beliefs  (Fink,  1995). 

Summary 

The  careful  construction  of  a  survey  directed  at  a  representative  population  can 
garner  valuable  information.  Effort  must  be  made  to  ensure  that  the  solicited  responses 
address  the  issues  queried.  If  this  is  the  case,  the  data  can  be  screened  in  terms  of 
frequency  and  means  to  produce  usable  information— The  Weight  Matrix.  The 
information  within  the  applicable  domain  (user  preferences  and  training  scenarios)  can 
then  be  manipulated  to  determine  the  relative  ranks  of  the  scenarios.  It  is  important  to 
determine  if  the  analysis  techniques  and  instrument  are  valid.  Table  5  summarizes  the 
efforts  in  the  literature  and  their  applications.  The  works  and  authors  are  compared  to 
their  area  of  primary  focus. 
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TABLE  5  LITERATURE  REVIEW 


Work 


Gazing  into  the  Oracle  _ _ 


Rules  of  the  Mind 


Training  Management _ 


How  Probable  is  Probable 


Sampling  Techniques _ 


Designing  Surveys _ 


Group  Techniques  for 

Program  Planning _ 


Battle  Focused  Training. 


Expert  Systems  Design 

and  Development _ 


The  Survey  Kit 


Proceedings  of  the  1991  Summer 
Computer  Simulation  Conference 


The  Secret  of  Future  Victories 


The  High  Level  Architecture  _ 


Design,  Development  and  Testing  of 
an  Automated  Knowledge-acquisition 
Tool  to  Aid  Problem  Solving,  Decision 
Making,  and  Planning  _ 


Advanced  Questionnaire  Design 


Practical  Methods  for  Design 

and  Analysis  of  Complex  Surveys 


Guide  for  the  Use  of  Models 
and  Simulations 


The  Value  of  Simulation  for  Training 


Systems  Acquisition  Manager’s _ 


Task  Performance  Support 

(TPS)  Codes  _ ___ 


The  Use  of  Computer-based 
Planning  to  Enhance  the 
Effectiveness  and  Efficiency 
of  Simulation-based  Team  Training 


Commander’s  Battle  Staff 


An  Overview  of  Knowledge 

Acquisition  Methods  _ 


Task  Analysis  _ __ 


The  PCC  Methodology  _ 


IUSARI 


Welbank 


|  Williams 


Bedell 


X  I  X  I  X  I  X  I  X  I  X 
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CHAPTER  5  INSTRUMENT  CONSTRUCTION  AND  VALIDATION 

Purpose 

The  U.S.  Army  currently  has  many  effective  tools  and  methodologies  for  the 
planning  of  live  simulation  training.  Literature  research  indicates  there  are  difficulties 
associated  with  virtual  and  constructive  simulation  training  planning,  where  as  excellent 
methodologies  exist  for  planning  live  simulation  training.  Further,  experts  indicate  that 
METT-T  and  other  factors  could  be  used  to  improve  the  virtual  and  constructive 
simulation  training  planning.  Hence,  the  purpose  of  the  survey  used  in  this  research  is  to 
determine  what  METT-T  factors  affect  the  selection  of  virtual  and  constructive 
simulation  training  scenarios.  Associated  with  this  purpose  are  two  sub  objectives:  1) 
Determine  if  the  level  of  importance  of  the  METT-T  factors  differs  between  virtual  and 
constructive  simulation  environments,  and  2)  Are  there  population  sub-groups  between 
which  there  are  further  differences? 

Additional  points  of  interest  include  determining  if  a  strong  correlation  between 
virtual  and  constructive  selection  factor  responses  exists  and  if  a  significant  correlation 
exists  between  selection  factors  within  virtual  and  constructive  environments. 
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Population 


The  survey  must  reach  the  appropriate  population  if  it  is  to  be  useful.  It  is 
important  to  define  the  frame  in  order  to  include  the  people  who  can  contribute  to  the 
area  of  interest  while  excluding  those  who  cannot.  In  order  to  qualify  to  take  this 
survey,  each  respondent  must  be,  or  have  been,  responsible  for  planning  training  for 
large  groups  of  soldiers,  understand  the  Army’s  current  doctrine,  and  understand  the 
Army’s  training  philosophy.  This  requirement  is  established  to  ensure  that  the 
respondents  are  experienced  in  planning  and  conducting  collective  training  and  are 
knowledgeable  as  to  the  appropriate  tactics,  techniques,  and  conditions  appropriate  to 
productive  training  within  their  specialty.  Explicitly,  in  order  to  qualify  to  respond  to  the 
survey  the  participant  must  meet  one  of  the  following  sets  of  criteria. 

•  A  U.S.  Army  officer,  rank  of  major  or  above,  who  has  successfully  completed  at 
least  50%  of  The  U.S.  Army’s  Command  and  General  Staff  School  or  its  Military 
Education  Level  (MEL)  IV  equivalent. 

•  A  U.S.  Army  officer,  rank  of  captain  or  above,  who  has  successfully  completed  a 
company  level  command,  and  currently  works  at  Tactics  or  Doctrine  Division  at  one 
of  the  Branch  schools. 

•  Any  U.S.  Army  officer  that  is  currently  serving  or  has  served  as  a  battalion  level  or 
above  commander  or  operations  officer  of  an  operational  unit  in  the  last  five  years. 
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A  survey  is  undertaken  in  six  stages.  Once  the  problem  is  identified  and  an 
objective  is  established  the  survey  must  be  designed,  pretested,  finalized,  distributed  and 
collected,  and  analyzed. 

There  is  no  Department  of  the  Army  requirement  for  operations  officers  to  have 
completed  Command  and  General  Staff  College.  However,  division  commanders  usually 
set  that  as  a  local  policy  (Mahanken,  Telephone  Interview,  March  1997).  It  is  further 
designated  that  operations  officers  are  responsible  for  ensuring  that  training  is  “oriented 
on  the  conditions  and  standards  of  combat”  (Combined  Arms  and  Service  Staff  School, 
Staff  Skills.  Roles  and  Relationships,  1991). 

The  independent  variables  are  collected  by  the  demographic  portion  of  the 
survey.  Theses  variables  are  specific  characteristics  of  the  subjects.  These  include  the 
facts  about  the  individual’s  training  history,  military  specialty,  and  their  training 
experiences 

There  are  two  sets  of  dependent  variables.  These  are  the  values  each  study 
participant  places  on  the  METT-T  factors  for  virtual  and  constructive  environments. 
These  measures  of  belief  are  collected  so  they  can  be  compared  to  the  independent 
variables  to  check  for  correlation.  A  detailed  explanation  of  the  purpose  and  utility  of 
each  question  is  in  Appendix  B. 
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Pretesting  the  Survey 


Preliminary 

The  original  survey  was  composed  of  eight  demographic  and  19  substantive 
questions  preceded  by  a  short  introduction  describing  the  purpose  of  the  survey.  (See 
Appendix  C.)  All  of  the  questions  were  designed  to  be  concrete  close  ended  questions 
with  ordinal,  nominative,  or  numerical  responses.  This  form  of  question  is  designed  to 
eliminate  any  ambiguity  in  the  meaning  of  the  question  and  confine  the  responses  to  a 
form  that  supports  later  mathematical  analysis  (Fink,  1996).  The  numerical  responses 
referred  to  values  such  as  age  and  military  education  level;  the  nominal  responses  applied 
to  branch  of  assignment;  and  the  ordinal  responses  covered  the  appropriateness  of 
simulators  that  the  respondent  has  used  in  the  past. 

The  substantive  portion  consisted  of  19  questions  where  the  respondent  is  asked 
to  indicate  how  greatly  each  factor  would  affect  their  selection  of  a  training  scenario 
(ordinal).  The  ordinal  form  used  for  all  of  these  questions  was  derived  from  work  done 
Ruth  Beyth-Marom  which  focused  the  responses  to  the  survey  into  a  form  that  eliminates 
ambiguity  and  fosters  analysis.  The  categorical  and  numeric  responses  support  both 
analysis  based  on  frequency  of  response  and  analysis  of  means  and  variance. 

In  this  form  the  survey  was  shown  to  three  observers  to  discuss  content, 
appearance,  and  approach.  This  preliminary  phase  of  pretesting  was  completed  to  ensure 
that  the  survey  appeared  professional  and  seemed  to  cover  the  correct  material. 
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Formal 


Formal  pretesting  of  the  survey  and  the  individual  questions  used  interview, 
overview,  and  sample  gathering  techniques.  Survey  pretesting  to  ensured  that  the 
questions  allowed  the  respondent  to  focus  on  the  issue,  that  there  were  no  social  or 
professional  desirability  associated  with  any  possible  response,  that  there  was  no  linkage 
to  authority,  that  all  complex  terms  were  defined,  and  that  the  dimensions  of  interest 
were  measured. 

The  interview  or  think-aloud  technique  consisted  of  separate  interviews  with 
active  duty  army  officers  ranked  captain  through  lieutenant  colonel  (7  in  total).  The 
selectees  belonged  to  the  same  frame  as  the  target  population.  This  technique  supports 
the  most  accurate  evaluation  of  the  survey  because  respondents  are  asked  to  read  each 
portion  (question  or  instruction)  and  relate  the  meaning  of  the  portion  to  the  interviewer 
(Czaja,  1996).  During  this  iterative  process  each  question  is  refashioned  so  that  the 
interpretation  of  the  question  is  matched  by  its  intent.  No  focus  groups  were  used  to 
examine  the  questions  because  their  use  could  obscure  problems  associated  with  a 
minority  of  the  group  (Fink,  How  to  Design  Surveys,  1995). 

An  overview  method  was  then  used  to  elicit  comments  on  the  survey  from 
academic  and  active  duty  military  personnel  involved  in  simulations.  In  the  overview 
process  the  respondent  comments  on  portions  of  the  survey  that  may  seem  ambiguous  or 
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unclear.  The  instructions  accompanying  the  survey  were  changed  based  on  the  feedback 
gained  in  this  phase. 

The  final  phase  of  the  pretesting  of  the  survey  was  the  giving  of  the  survey  to  a 
group  of  respondents  to  generate  data  to  be  used  to  test  the  data  analysis  techniques.  This 
was  done  so  the  appropriateness  of  the  planned  statistical  analysis  techniques  could  be 
tested  before  the  survey  was  distributed  to  the  target  population.  This  exercise  produced 
essentially  undifferentiated  data.  This  was  caused  because  the  pretest  sample  did  not 
have  (and  was  not  expected  to  have)  the  different  characteristics  that  would  have 
revealed  different  responses. 

The  survey’s  final  form  is  as  it  appears  in  Appendix  D. 

Distribution 


The  survey  was  sent  to  selected  active  duty  and  reserve  brigade  and  battalion 
commanders,  operations  officers,  and  other  qualified  respondents.  The  emphasis  was  to 
elicit  responses  from  those  units  making  the  greatest  use  of  simulations.  The  survey  went 
to  service  branch  schools  to  be  evaluated  by  their  respective  Tactics  Division  and 
Doctrine  Division.  All  respondents  were  pre-contacted  to  insure  a  high  rate  of  return. 


Analysis 
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The  common  methods  of  analyzing  survey  data  include  means,  medians,  modes, 
ranges,  standard  deviations,  and  analysis  of  variance  (ANOVA)  (Fink,  1995).  In  surveys 
comparing  nominal  independent  variables  and  Ordinal  or  numerical  dependent  variables 
it  is  customary  to  conduct  the  analysis  using  an  ANOVA  technique  (Fink,  1995).  It  is 
also  possible  to  use  regression  techniques  or  frequency  analysis. 

The  analysis  conducted  on  this  survey  is  accomplished  using  means,  an  ANOVA 
technique  with  multiple  comparisons,  pair  wise  comparisons  using  z  or  t  test  statistics, 
and  a  Chi  Squared  test  for  independence  to  determine: 

1 .  If  for  the  entire  sample  the  factors  have  an  affect  on  the  selection  of  the 

training  scenarios. 

2.  If  there  are  factors  that  are  significant  in  one  simulation  environment  and  not  the 
other. 

3.  If  certain  selection  factors  are  more  important  than  other  factors  in  a  given 

environment. 

4.  If  different  sample  sub-groups  select  different  factors  as  being  significant. 

An  examination  of  factor  means  will  determine  which  factors  are  significant. 

Any  factor  whose  value  (including  the  confidence  interval)  is  less  than  or  equal  to  30 
(has  a  small  chance  of  affecting  the  selection  on  the  Beyth-Marom  scale)  will  be 
discounted  as  not  important. 
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The  Fisher  technique,  a  multiple  comparison  ANOVA,  is  used  to  measure  the 
variation  in  the  dependent  variables.  The  variables’  confidence  intervals  around  their 
means  will  be  compared  when  the  they  are  grouped  around  the  independent  variables 
such  as  simulation  environment  (virtual  versus  constructive)  or  branch  association.  This 
test  uses  pooled  standard  deviations  so  that  it  compares  each  mean  to  the  population  as  a 
whole. 

The  fourth  group  of  tests  will  be  done  by  examining  the  means  within  and 
between  each  category.  This  will  be  done  using  a  simple  comparison  of  the  means  using 
the  central  limit  theorem  and  the  equation: 

.  (5~  *2  )-D0 

V  nX  n2 

(equation  4) 


Where 


z  =  the  test  statistic 

x,  =  the  mean  from  the  first  sample 

x2=  the  mean  from  the  second  sample 

D0=  the  hypothesized  difference  between  the  two 

means 

a  j  =  the  variance  of  the  first  sample 
a  j  =  the  variance  from  the  second  sample 
n=  the  number  of  points  in  the  first  sample 
n2  =  the  number  of  points  in  the  second  sample 
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Because  n  >  30  the  central  limit  theorem  holds  so  there  is  no  assumption  of  normalcy 
required  for  this  test  (Malone,  Personal  Interview,  1996).  This  test  compares  each  mean 
against  each  other. 

The  simulation  environment  test  results  will  be  compared  to  similar  tests  for 
independence  using  Chi  Squared  techniques.  The  analysis  of  expected  frequency  tables 
will  support  the  determination  of  whether  frequency  is  independent  of  the  population 
group  or  the  environment. 
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Analysis  of  Pretest  Survey  Data 


The  tabulated  data  from  the  30  pretest  surveys  appears  in  Appendix  E.  The 
pretest  group  consisted  of  ROTC  cadets.  The  survey  was  administered  to  the  cadets  as 
part  of  their  training  and  all  surveys  were  returned. 

Table  6  contains  all  of  the  mean  values  and  lists  them  in  descending  order.  A 
simple  means  analysis  of  the  factors  indicates  that  all  of  the  factors  are  important  to  the 
pretest  participants  in  selecting  a  simulation  training  scenario.  Since  all  of  the  factors 
appear  significant  in  Table  5  according  to  the  screening  criteria  there  is  some  question  as 
to  the  validity  of  the  test.  These  results  could  accurately  measure  the  importance  of  all  of 
the  factors  or  could  inaccurately  report  some  or  all  of  the  factors  as  important  when  they 
truly  were  not  important.  They  selection  factors  were  selected  because  of  their  perceived 
importance  and  so  it  is  determined  that  this  result  confirms  that  all  of  the  factors  are 
important  to  selecting  simulation  training  scenarios. 
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TABLE  6  PILOT  FACTOR  MEANS 


Constructive 

Mean 

Virtual 

Mean 

Exercise  Preparation 

63.90 

Terrain 

75.73 

Combat  Power  Ratio 

63.57 

Weather 

73.03 

Enemy  Equipment 

62.77 

Previous  Use 

70.33 

Enemy  Task  Organization 

61.43 

Friendly  Equipment 

69.40 

Enemy  Composition 

61.33 

Level  Difficulty 

68.87 

Enemy  Mission 

56.43 

Simulator 

66.83 

Enemy  Training  Level 

56.30 

Unit  Assessment 

66.77 

Task 

54.33 

Combat  Power  Ratio 

65.43 

Previous  Use 

54.17 

Enemy  Composition 

64.43 

Observation  Devices 

54.04 

Mission  Planning 

62.70 

Level  Difficulty 

53.03 

Observation  Devices 

62.40 

Mission  Planning 

52.77 

Enemy  Training  Level 

62.10 

Light  Data 

51.23 

Enemy  Equipment 

62.03 

Weather 

48.73 

Friendly  Composition 

58.90 

Friendly  Equipment 

47.60 

Light  Data 

57.97 

Terrain 

47.00 

Adjacent  Unit 

57.47 

Adjacent  Unit 

46.03 

Enemy  Mission 

56.93 

Unit  Assessment 

44.80 

Task 

53.17 

Friendly  Composition 

43.67 

Exercise  Preparation 

51.80 

Simulator 

42.47 

Enemy  Task  Organization 

51.53 

(Computer  output  is  located  in  Appendix  F.) 


The  next  item  of  interest  is  to  determine  if  the  pretest  survey  could  be  used  to 
determine  if  selection  factors  could  have  various  levels  of  importance  based  on  the 
simulation  environment.  Based  on  the  95%  confidence  intervals  (Cl)  for  the  factor 
means  using  the  pooled  standard  deviation,  there  are  several  factors  that  that  have 
different  levels  of  importance  based  on  their  association  with  virtual  or  constructive 
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simulations.  For  example,  the  factor  ‘Terrain’  in  a  virtual  context  has  a  95%  chance  of 
being  between  80  and  70.  Terrain  in  a  constructive  context  has  a  95%  chance  of  being 
between  57  and  37.  Because  these  two  confidence  intervals  do  not  over  lap  then  the  two 
factors’  values  cannot  be  the  same  at  a  95%  confidence  level.  Further  examination  yields 
different  levels  of  importance  for  factors  based  on  environment  for  “simulator 
selection”,”  friendly  equipment”,”  weather”,  and  “observation  devices”  (See  Appendix 

F) . 

Determining  the  ability  of  the  instrument  to  measure  the  different  levels  of 
importance  of  selection  factors  within  a  single  environment  is  conducted  next.  A 
comparison  of  the  means  of  Observation  Devices  (OD)  and  Combat  Power  Ratios  (CPR) 
was  conducted  using  the  z  statistic  test  in  the  constructive  environment  (See  Appendix 

G) .  This  test  showed  that  the  mean  of  the  CPR  was  significantly  larger  than  that  of  OD. 
Though  no  normalcy  assumption  was  made  using  this  test,  each  sample  was  found  to 
follow  a  normal  distribution  using  a  chi-squared  test  (See  Appendix  G).  The  survey 
therefore  has  sufficient  granularity  to  compare  mean  values  within  an  environment. 

A  similar  comparison  of  armor  (Appendix  H)  and  infantry  (Appendix  I)  branch 
officers  (Branch  was  assigned  randomly  for  the  pilot  data)  reveals  no  statistical 
difference.  All  of  the  factors  have  overlapping  intervals.  This  result  was  expected 
because  the  branch  groups  were  randomly  assigned.  Therefore  the  population  groups 
should  not  have  had  distinguishing  characteristics. 
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The  factor  results  were  also  examined  using  a  chi  squared  test  for  independence. 
These  tests  were  done  to  because  they  did  not  rely  on  the  assumptions  associated  with 
parrametric  statistics.  The  survey  results  were  tabulated  using  the  frequency  of 
occurrence  of  each  category  (Very  Small  Chance,  Small  Chance,  Could  Effect,  High 
Chance  ,  and  Very  High  Chance).  The  resulting  frequency  tables  appear  in  Appendix  J. 

Table  7  shows  the  frequency  of  each  response^  occurrence  for  the  factor  Terrain 
in  each  environment.  The  chi-squared  test  lends  support  for  the  ANOVA  technique  used 
above.  The  level  of  significance  (p)  from  the  chi-squared  test  for  independence  indicates 
that  the  expected  frequencies  are  different  for  Terrain  depending  on  the  environment. 
This  is  in  agreement  with  the  results  of  the  ANOVA. 

TABLE  7  FREQUENCY  COMPARISON  BETWEEN  CONSTRUCTIVE  AND 

VIRTUAL 

FACTOR:  TERRAIN 


Factor 

Terrain 

Very  Small 
Chance 

Small 

Chance 

Could  Effect 

High 

Chance 

Very  High 
Chance 

Virtual 

1 

2 

1 

9 

17 

Constructive 

9 

4 

11 

3 

3 

p=0.000 

Table  8  contains  the  results  of  the  chi-squared  test  for  the  selection  factor  Task. 
The  large  value  of  “p”  indicates  that  the  value  associated  “Task”  is  unaffected  by 
changing  environments.  This  supports  the  ANOVA  findings  that  “Tasks”  level  of 
importance  unaffected  by  the  choice  of  environment  (See  Appendix  F). 
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TABLE  8  FREQUENCY  COMPARISON  BETWEEN  CONSTRUCTIVE  AND 

VIRTUAL 

FACTOR:  TASK 


Factor 

Task 

Very  Small 
Chance 

Small 

Chance 

Could  Effect 

High 

Chance 

Very  High 
Chance 

Virtual 

3 

3 

12 

9 

3 

Constructive 

2 

9 

9 

8 

2 

p=0.422 

Summary 

The  result  of  pre-testing  the  survey  is  the  determination  that  the  survey  and  the 
accompanying  analysis  techniques  are  valid.  The  pre-testing  of  the  survey  indicates  that 
it  should  be  able  to  determine: 


1 .  If  for  the  entire  sample  the  factors  have  an  affect  on  the  selection  of  the 

training  scenarios. 

2.  If  there  are  factors  that  are  significant  in  one  simulation  environment 

and  not  the  other. 

3 .  If  certain  selection  factors  are  more  important  than  other  factors  in  a  given 

environment. 

4.  If  different  sample  sub-groups  select  different  factors  as  being  significant. 

5 .  The  value  of  the  Weight  Matrix  can  be  determined 

It  is  now  appropriate  to  apply  the  instrument  to  the  target  population. 
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CHAPTER  6  DATA  ANALYSIS 


Responses 

Eighty-five  surveys  were  distributed  of  which  79  surveys  were  returned  for  the 
study.  Of  these  79,  62  were  qualified.  6  surveys  were  not  returned.  The  non-replies 
were  continually  re-contacted  concerning  the  missing  surveys.  The  previously  agreeable 
survey  candidates  stated  they  had  already  returned  the  survey  (not  received)  and  were  not 
interested  in  duplicating  their  previous  responses  (2)  or  that  the  survey  was  no  longer  a 
priority  for  them  and  they  could  not  participate  (4).  It  was  determined  that  7  more 
surveys  could  not  be  used  because  the  solicited  participant  had  another  member  of  their 
unit  complete  the  survey  (ranked  captain  through  private  first  class).  Twelve  of  the 
respondents  felt  unqualified  to  estimate  the  impact  the  factors  would  have  on  determining 
training  scenario  selection  in  the  virtual  environment.  The  final  response  rate  was  93%. 

The  participants  qualifications  and  backgrounds  covered  all  of  the  officer 
branches  and  ranged  in  rank  from  captain  (assigned  to  a  branch  school  and  therefore 
qualified)  to  promotable  colonel. 

Tables  26  and  27  summarize  the  population  membership  information.  In  Table 
26  the  affiliation  is  by  branch,  the  basic  category  of  assignment.  In  Table  27  the 
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population  is  divided  into  two  groups  Ground  Maneuver  and  Combat  Support  and 
Combat  Service  Support. 


TABLE  26  NUMBER  OF  RESPONDENTS  BY  CATEGORY 


Branch  or  Group 

Number  of 
responses 

Air  Defense 

4 

Armor 

8 

Aviation 

6 

Chemical  Corps 

1 

Corps  of  Engineers 

1 

Field  Artillery 

7 

Infantry 

16 

Military  Intelligence 

1 

Military  Police 

5 

Ordinance 

3 

Quarter  Master 

3 

Signal  Corps 

2 

Transportation  Corps 

1 

Undeclared 

4 

Total 

62 

TABLE  27  GROUND  MANEUVER  AND  CS  AND  CSS  RESPONDENTS 


Branch  or  Group 

Number  of 
responses 

Ground  Maneuver 

24 

Combat  Support  and  Combat  Service  Support  (CS  and  CSS) 

21 

Total 

45 

The  categories  “Ground  Maneuver”  and  “CS  and  CSS”  were  constructed  from  the 
responses.  Ground  Maneuver  consists  of  the  Armor  and  Infantry  data  while  CS  and  CSS 
is  made  up  from  the  data  associated  with  Air  Defense,  Chemical  Corps,  Military  Police, 
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Ordinance,  Quarter  Master,  Signal  Corps,  and  Transportation  Corps.  The  Undeclared 
category  is  made  up  of  the  respondents  who  elected  not  to  enter  a  branch.  The  complete 
set  of  tabulated  data  can  be  found  in  Appendix  K.  In  the  appendix  the  data  is  presented 
as  the  entire  population  and  broken  down  into  each  group. 


Tests  and  Analysis 

The  Determination  of  the  Mean 

The  arithmetic  mean  of  all  of  the  values  assigned  to  each  factor  were  determined. 
Each  of  these  values  was  screened  against  30,  the  pre-selected  level  of  minimal  concern, 
all  of  the  means  exceeded  this  value.  Table  28  contains  all  of  the  mean  values  and  lists 
them  in  descending  order. 
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TABLE  28  FACTOR  MEANS 


Constructive 

Mean 

Virtual 

Mean 

Light  Data 

77.80 

Light  Data 

75.70 

Previous  Use 

70.40 

Task 

66.40 

Task 

68.30 

Previous  Use 

65.70 

Simulator 

67.00 

Terrain 

65.50 

Exercise  Preparation 

66.80 

Friendly  Equipment 

64.40 

Level  Difficulty 

66.30 

Level  Difficulty 

63.10 

Friendly  Equipment 

66.00 

Simulator 

62.80 

Enemy  Composition 

65.90 

Enemy  Composition 

61.60 

Enemy  Mission 

65.70 

Unit  Assessment 

61.40 

Friendly  Composition 

65.48 

Enemy  Mission 

60.20 

Combat  Power  Ratio 

64.70 

Exercise  Preparation 

58.80 

Terrain 

64.50 

Weather 

58.70 

Mission  Planning 

62.60 

Combat  Power  Ratio 

58.40 

Enemy  Equipment 

58.50 

Mission  Planning 

57.80 

Enemy  Task  Organization 

58.00 

Observation  Devices 

57.00 

Adjacent  Unit 

57.70 

Enemy  Equipment 

55.80 

Enemy  Training  Level 

51.80 

Adjacent  Unit 

55.10 

Weather 

51.50 

Friendly  Composition 

53.60 

Observation  Devices 

49.60 

Enemy  Training  Level 

51.70 

Unit  Assessment 

48.30 

Enemy  Task  Organization 

51.40 

(See  Appendix  L) 


Analysis  of  Variance 

A  One  Way  Analysis  of  Variance  was  conducted  in  order  to  determine  which 
factor’s  levels  of  probable  impact  on  scenario  selection  were  significantly  larger  or 
smaller  than  others.  This  analysis  was  conducted  on  the  various  population  groupings 
and  is  summarized  in  Tables  29  through  35.  The  ANOVA  revealed  that  some  factors  are 
more  important  than  others  and  that  this  varies  according  to  population  group. 
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TABLE  29  ANOVA  RESULTS:  ALL  RESPONDENTS 


Population: 

All  Responses 

Most  Important  Factors: 

Constructive  Unit  Assessment 

Virtual  Unit  Assessment 

Constructive  Previous  Use  of  a  Scenario 

Least  Important  Factors: 

Constructive  Light  Data 

Constructive  Observation  Devices 

Virtual  Enemy  Training  Level 

(See  Appendix  L) 


TABLE  30  ANOVA  RESULTS:  ARMOR 


Population: 

Armor 

Most  Important  Factors: 

Virtual  Unit  Assessment 

Constructive  Unit  Assessment 

Least  Important  Factors: 

Constructive  Light  Data 

Constructive  Weather 

(See  Appendix  N) 


TABLE  31  ANOVA  RESULTS:  AVIATION 


Population: 

Aviation 

Most  Important  Factors: 

Constructive  Exercise  Preparation 

Virtual  Observation  Devices 

Least  Important  Factor: 

Constructive  Observation  Devices 

(See  Appendix  O) 
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TABLE  32  ANOVA  RESULTS:  INFANTRY 


Population: 

Infantry 

Most  Important  Factors: 

Constructive  Task 

Constructive  Terrain 

Least  Important  Factor: 

Virtual  Exercise  Preparation 

Constructive  Observation  Devices 

Virtual  Enemy  Training  Level 

TABLE  33  ANOVA  RESULTS:  MILITARY  POLICE 


Population: 

Military  Police 

Most  Important  Factors: 

Constructive  Unit  Assessment 

Virtual  Unit  Assessment 

Constructive  Previous  Use  of  a  Scenario 

Least  Important  Factors: 

Virtual  Terrain 

Virtual  Observation  Devices 

Virtual  Light  Data 

(See  Appendix  R) 
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TABLE  34  ANOVA  RESULTS:  GROUND  MANEUVER 


Population: 

Ground  Maneuver 

Most  Important  Factors: 

Constructive  Unit  Assessment 

Virtual  Unit  Assessment 

Virtual  Terrain 

Least  Important  Factors: 

Constructive  Weather 

Constructive  Observation  Devices 

Constructive  Enemy  Training  Level 

(See  Appendix  T) 


TABLE  35  ANOVA  RESULTS:  COMBAT  SUPPORT  AND  COMBAT  SERVICE 

SUPPORT 


Population: 

Combat  Support  and  Combat  Service 
Support 

Most  Important  Factors: 

Constructive  Unit  Assessment 

Virtual  Unit  Assessment 

Constructive  Previous  Use  of  a  Scenario 

Least  Important  Factors: 

Virtual  Enemy  Training  Level 

Virtual  Observation  Devices 

Virtual  Enemy  Training  Level 

(See  Appendix  U) 
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There  were  no  factors  significantly  more  likely  to  be  considered  for  Air  Defense, 
Chemical  Corps,  Corps  of  Engineers,  Field  Artillery,  Ordinance,  Quarter  Master,  Signal 
Corps,  or  Transportation  Corps.  This  is  could  be  a  result  of  the  small  number  of 
participants  or  because  no  factors  are  more  significant  than  any  others. 


Refinement  of  the  One  Way  ANOVA  of  the  Combined  Population 

Many  of  the  results  are  very  close  in  value  based  on  the  confidence  intervals 
around  the  means.  Further  examination  of  the  factors  can  be  done  using  a  two  tailed  z 
statistic  test  for  the  difference  between  two  means.  Prior  to  the  conducting  this  test  it 
may  be  important  to  recognize  the  relationships  between  the  variances.  An  F  test  of 
equal  variances  can  determine  this  relationship.  This  relationship  then  ensures  that  the 
appropriate  mean  test  is  conducted. 


Where 

s\  =  the  variance  of  population  1 
si  =  the  variance  of  population  2 


ff numerator  til  1 

W "denominator  ^  1 


(equation  5) 
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The  results  of  this  analysis  are  shown  in  Table  36.  The  table  shows  that  only  3 


factors’  means  are  equal  to  the  Constructive  Terrain  Mean. 


TABLE  36  REFINEMENT  OF  ANOVA:  CONSTRUCTIVE  TERRAIN 


Factor  Mean  is  Less  than 
the  Constructive  Terrain 
Mean 

Factor  Mean  is  Equal  to 
the  Constructive  Terrain 
Mean 

Factor  Mean  is  More 
than  the  Constructive 
Terrain  Mean 

Constructive  Enemy 
Training  Level 

Constructive  Enemy  Task 
Organization 

Constructive  Weather 

Virtual  Enemy  Equipment 

Constructive  Light  Data 

Virtual  Adjacent  Unit 

Constructive  Observation 
Devices 

Virtual  Friendly 
Composition 

Virtual  Enemy  Task 
Organization 

Virtual  Enemy  Training 
Level 

Comparing  the  Importance  of  the  Constructive  and  Virtual  Factors 

A  two  tailed  z  statistic  test  for  the  difference  between  two  means  was  conducted 
to  determine  the  relative  impact  of  Virtual  and  Constructive  aspects  of  each  factor  for  the 
total  population.  The  surveys  that  did  not  include  responses  for  each  environment  were 
eliminated  from  this  analysis.  The  critical  value  for  z  in  all  of  the  tests  was  1.96  (1.9599) 
at  a  0.05  level  of  significance.  Any  test  that  produced  a  z  statistic  for  which  the  absolute 
value  exceeded  1 .96  was  determined  to  reject  the  hypothesis  that  the  means  were  equal. 
Table  37  summarizes  the  results  of  the  analysis.  The  environmental  means  were  only 
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different  for  Light  Data  and  Observation  Devices  .  All  results  can  be  found  in  Appendix 


V. 


TABLE  37  Z  TEST:  CONSTRUCTIVE  VERSUS  VIRTUAL 


Means  Are  Not 
Equal 

Means  are  Equal 

Light  Data 

Terrain 

Observation  Devices 

Task 

Simulator 

Friendly  Composition 

Enemy  Composition 

Combat  Power  Ratio 

Enemy  Task  Organization 

Enemy  Equipment 

Friendly  Equipment 

Adjacent  Unit 

Enemy  Training  Level 

Enemy  Mission 

Weather 

Previous  Use 

Level  Difficulty 

Unit  Assessment 

Exercise  Preparation 

Mission  Planning 

Though  this  type  of  test  can  be  completed  on  any  two  factor  value  sets,  an  exhaustive 
analysis  would  require  2380  different  combinations  of  factors  and  categories.  All  of  the 
test  results  can  be  found  in  Appendix  W. 
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Test  for  Independent  Frequencies 


All  of  the  numerical  responses  given  to  each  factor  were  categorized  to  produce 
Table  38.  The  table  lists  each  factor  and  the  number  of  responses  for  each  category 
(Very  Small  Chance  through  Very  High  Chance): 
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TABLE  38  FACTOR  FREQUENCIES  FOR  ALL  RESPONDENTS 


Very  Small 
Chance 

Small 

Chance 

Could 

Effect 

High 

Chance 

Very  High 
Chance 

Factor 

C  Terrain 

6 

7 

10 

24 

15 

C  Task 

2 

6 

17 

22 

15 

C  Simulator 

5 

9 

8 

21 

17 

C  F  Comp 

8 

6 

12 

17 

19 

C  E  Comp 

6 

10 

9 

13 

24 

C  Power  Ratio 

6 

5 

15 

17 

19 

C  Enemy  Task 

Org 

12 

6 

11 

14 

19 

C  E  Equip 

9 

8 

17 

11 

17 

4 

7 

17 

15 

19 

C  Adjacent  Unit 

8  [ 

8 

22 

11 

13 

C  E  Train  Level 

13 

10 

15 

14 

10 

C  E  Mission 

6 

7 

11 

18 

20 

C  Weather 

13 

11 

16 

10 

12 

C  Previous  Use 

2 

5 

16 

19 

20 

ll  II  1  II  M 

5 

5 

17 

13 

21 

C  Assessment 

1 

4 

8 

19 

29 

C  Light  Data 

18 

5 

16 

13 

10 

C  Observation 
Devices 

16 

12 

14 

10 

10 

C  Exercise  Prep 

6 

8 

7 

21 

20 

C  Mission 
Planning 

9 

9 

9 

14 

21 

V  Terrain 

6 

4 

8 

15 

17 

V  Task 

3 

7 

11 

14 

15 

V  Simulator 

7 

6 

9 

11 

16 

11 

9 

9 

9 

12 

V  E  Comp 

5 

10 

8 

13 

14 

V  Power  Ratio 

5 

7 

18 

11 

9 

V  Enemy  Task 

Org 

12 

6 

12 

9 

11 

V  E  Equip 

10 

4 

12 

14 

10 

Very  Small 
Chance 

Small 

Chance 

Could 

Effect 

High 

Chance 

Very  High 
Chance 

V  F  Equip 

4 

6 

12 

15 

13 

V  Adjacent  Unit 

7 

10 

13 

13 

7 

V  E  Train  Level 

13 

7 

9 

11 

10 

Factor 

V  E  Mission 

8 

7 

8 

14 

13 

V  Weather 

9 

3 

16 

9 

13 

V  Previous  Use 

3 

9 

10 

14 

14 

V  Difficulty 

5 

7 

13 

10 

15 

V  Assessment 

5 

3 

4 

9 

28 

V  Light  Data 

10 

4 

14 

10 

12 

V  Observation 
Devices 

8 

6 

11 

9 

16 

9 

10 

6 

11 

14 

V  Mission 
Planning 

10 

7 

10 

9 

14 

A  chi-squared  test  for  independence  was  then  conducted  to  determine  if  the 
frequency  that  each  factor  was  considered  to  be  of  a  particular  importance  was 
independent  of  the  environment  under  which  it  is  considered.  At  the  0.05  level  of 
significance  none  of  the  factors  frequencies  were  effected  by  the  environment.  (Note  the 
factor,  Unit  Assessment,  could  not  be  evaluated  because  more  than  20%  of  the  expected 
values  were  less  than  five  (5).  (All  of  the  chi  squared  test  results  are  in  Appendix  X.) 

The  same  expected  frequency  sparseness  problem  arose  with  the  examination  of 
factor  response  frequency  when  comparing  all  of  the  factors,  virtual  and  constructive,  in 
terms  of  Ground  Maneuver  and  CS  and  CSS.  (See  Ground  Maneuver  and  CS  and  CSS 
Frequency  Table  in  Appendix  Y.)  All  of  the  resulting  calculations  produced  more  than 
20%  expected  values  under  five  (5).  (See  Appendix  Z.) 
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In  an  effort  to  reduce  the  sparseness  of  the  expected  values  the  “Very  Small 
Chance”  and  the  “Small  Chance”  frequencies  were  combined  into  “Small”.  Similarly, 
“High  Chance”  and  “Very  High  Chance”  were  combined  into  “High”.  The  New 
frequency  table  appears  in  Appendix  Z  along  with  the  frequency  independence  tests. 

Even  after  combining  the  ranges,  many  of  the  expected  values  remained  less  than  five  (5) 
and  the  factor’s  frequencies  remained  independent  of  the  grouping,  Ground  Maneuver 
versus  Combat  Service  and  Combat  Service  Support.  Table  39  summarizes  these  results. 

TABLE  39  FREQUENCY  TEST:  GROUND  MANEUVER  VERSUS  COMBAT 
SUPPORT  AND  COMBAT  SERVICE  SUPPORT 


Factor  frequency  is  independent 
of  Ground  Maneuver  or  CS  & 

CSS 

More  than  20%  of  the  expected  values  are 
sparse  (>5) 

Constructive 

Constructive 

Terrain 

Task 

Simulator 

Combat  Power  Ratio 

Friendly  Composition 

Enemy  Equipment 

Enemy  Composition 

Friendly  Equipment 

Enemy  Task  Organization 

Adjacent  Unit 

Enemy  Mission 

Enemy  Training  Level 

Previous  Use 

Weather 

Difficulty 

Light  Data 

Assessment 

Observation  Devices 

Exercise  Preparation 

Mission  Planning 

Virtual 

Virtual 

Terrain 

Combat  Power  Ratio 

Task 

Enemy  Task  Organization 

Simulator 

Enemy  Equipment 

Friendly  Composition 

Adjacent  Unit 

Enemy  Composition 

Light  Data 

Friendly  Equipment 

Enemy  Training  Level 
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Factor  frequency  is  independent 
of  Ground  Maneuver  or  CS  & 

CSS 

More  than  20%  of  the  expected  values  are 
sparse  (>5) 

Enemy  Mission 

Weather 

Previous  Use 

Difficulty 

Assessment 

Observation  Devices 

Exercise  Preparation 

Mission  Planning 

Assigning  Weights  to  the  Scenario  Selection  Factors 


Two  potential  techniques  exist  for  assigning  weights  to  the  various  scenario 
selection  factors.  The  first  technique  requires  an  exhaustive  series  of  pair-wise 
comparisons  of  the  factor  means  and  variances.  This  method  would  result  in  groups  of 
factors  whose  means  could  not  be  proven  to  be  statistically  different  at  a  0.05  level  of 
significance.  These  factor-groups  would  then  be  assigned  a  weight  or  value  derived  from 
the  means  of  all  of  the  members  of  the  group.  This  methodology  would  become  quite 
cumbersome. 

A  far  better  approach  is  to  address  the  issue  of  factor  weights  through  the 
frequency  table.  The  modal  frequency  group  would  determine  the  weight  associated 
with  the  selection  factor. 

Table  40  is  constructed  by  combining  all  of  the  frequency  response  data  for 
environmental  (constructive  and  virtual)  and  population  (Ground  Maneuver  and  Combat 
Service  and  Combat  Service  Support)  groups.  This  can  be  done  because  the  frequency 
analysis  showed  the  responses  to  be  independent  of  these  associations. 
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TABLE  40  COMBINED  CONSTRUCTIVE  AND  VIRTUAL  FREQUENCY 

TABLE 


Very  Small 
Chance 

Small 

Chance 

Could 

Effect 

High 

Chance 

Very  High 
Chance 

Factor 

Terrain 

12 

11 

18 

39 

32 

Task 

5 

13 

28 

36 

30 

Simulator 

12  ! 

15 

17 

32 

33 

Friendly  Composition 

19 

15 

21 

26 

31 

Enemy  Composition 

11 

20 

17 

26 

38 

Combat  Power  Ratio 

11 

12 

33 

28 

28 

Enemy  Task 
Organization 

24 

12 

23 

23 

30 

Enemy  Equipment 

19 

12 

29 

25 

27  ] 

Friendly  Equipment 

8 

13 

29 

30 

32 

Adjacent  Unit 

15 

18 

35 

24 

20 

Enemy  Training  Level 

26 

17 

24 

25 

20 

Enemy  Mission 

14 

14 

19 

32 

33 

Weather 

22 

14 

32 

19 

25 

Previous  Use 

5 

14 

26 

33 

34 

Difficulty 

10 

12 

30 

23 

36 

Assessment 

6 

7 

12 

28 

57 

Light  Data 

28 

9 

30 

23 

22 

Observation  Devices 

24 

18 

25 

19 

26 

Exercise  Prep 

15 

18 

13 

32 

34 

Mission  Planning 

19 

16 

19 

23 

35 

Table  41  is  generated  by  selecting  the  modal  value  from  Table  40. 


TABLE  41  FACTOR  WEIGHTS 


Very  Small 
Chance  (.1) 

Small 

Chance  (.3) 

Could 
Effect  (.5) 

High 

Chance(.7) 

Very  High 
Chance(.9) 

Factor 


Terrain 


Task 


Simulator 


Friendly  Composition 


Enemy  Composition 


Combat  Power  Ratio 


Enemy  Task 
Organization 


Enemy  Equipment 


Friendly  Equipment 


Adjacent  Unit 


Enemy  Training  Level 


Enemy  Mission 


Weather 


Previous  Use 


Difficulty 


Assessment 


Light  Data 


Observation  Devices 


Exercise  Prep 


Mission  Planning 
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.7 

.7 

.9 

.9 

.9 

.5 

.9 

.5 

.9 

.5 

.1 

.9 

.5 

.9 

.9 

.9 

.5 

.9 

.9 

.9 


FIGURE  1:  THE  WEIGHT  MATRIX 

The  Weight  matrix  is  the  final  result  of  the  survey  data  and  analysis.  All  of  the 
collected  data  is  processed  to  produce  this  matrix  that  supports  the  multi-dimensional 
(METT-T  factors)  analysis  of  multiple  simulation  training  scenarios. 
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Use  of  the  System  by  a  Potential  User 


As  an  example  use  of  the  system,  consider  the  following:  a  user  with  the 
randomly  assigned  preferences,  listed  in  Table  42,  must  decide  between  virtual 
simulation  training  scenario  1  (STS  1)  or  virtual  simulation  training  scenario  2  (STS  2)  . 
The  preferences  in  Table  42  are  drawn  from  Tables  9  through  25  in  chapter  3. 


User  Preference  Matrix 

TABLE  42  EXAMPLE  USER  PREFERENCES 


Category 

Desired  Value 

UPC  Value 

Particular  task  to  be  trained 

Task  number  1642 

See  Task  matrix 

Combat  Power  Ratio 

Doctrinal 

2 

Enemy  Composition 

Tank  Company 

9 

Enemy  Task  Organization 

Advanced  Guard 

4 

Enemy  Equipment 

No  preference 

0 

Enemy  Training  Level 

Standard 

2 

Enemy  Mission 

Attack 

6 

Exercise  Preparation  Time 

Complete  package 

2 

Mission  Planning  Time 

No  Preference 

0 

Terrain 

Desert 

1 

Weather 

Clear 

6 

Light  Data 

Dusk 

4 

Friendly  Composition 

Battalion 

12 

Friendly  Equipment 

Current 

2 

Adjacent  Unit 

U.S.  Army 

10 

Supported  Observation  Devices 

Thermal  and  Optical 

2 

Previous  Use  of  a  Scenario 

No  previous  use 

See  Previous  Use  matrix 

Level  of  Difficulty 

Standard 

2 

Simulator 

No  Preference 

0 

Unit  Assessment 

Yes 

See  Untrained  matrix 
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Table  42  is  in  reality  a  multi-dimensional  table.  For  example,  in  order  to 
determine  [Un]  one  must  use  the  Task  matrix  and  the  Scenario  Task  matrix.  The 
intersection  of  matrices  yields  [Un]. 


Task  Matrix 

In  the  military,  tasks  are  generally  represented  by  a  Mission  Training  Plan  (MTP) 
number  (Training  Management  1991).  For  this  example  1642  represents  the  task 
“Occupy  an  Assembly  Area”.  Since  our  fictional  user  desires  to  train  his  unit  on  task 
1642,  that  task  number  is  placed  in  the  Task  matrix,  [ty].  The  various  tasks  that  are 
trained  by  the  Simulation  Training  Scenarios  (STS)  appear  in  the  respective  Scenario 
Task  matrices,  [Ty]. 

The  value  of  [UX1]  is  determined  by  the  intersection  of  the  Task  matrix,  [ty].  with 
the  Scenario  Task  matrix  [Ty]  for  each  scenario. 


’l352' 

1352~ 

[tij]  =  [1642] 

[Ty]  for  STS  1  = 

1642 

5429 

[Ty]  for  STS  2  = 

1142 

243  1 

The  values  for  [Uu]  for  STS  1  is  0;  the  value  for  STS  2  is  1 .  This  indicates  that 
STS  1  contains  the  desired  task  while  STS  2  does  not. 

Having  determined  the  value  of  [Uu],  we  must  now  consider  the  users  desire  not 
to  repeat  the  use  of  a  training  scenario.  This  phase  determines  the  value  of  [Ut  17] 
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Previous  Use  Matrix 


The  value  of  [U117]  is  determined  by  the  relationship  between  the  Previous  Use 
matrix,  [Pij]  and  the  scenario  number.  The  Previous  Use  matrix,  [Py]  contains  all  of  the 
numbers  of  the  training  scenarios  the  unit  has  previously  executed.  Because  the  user 
preferred  not  to  reuse  a  scenario  (see  Table  42),  the  intersection  of  the  sets  are  examined. 
If  the  intersection  is  the  null  set  then  Uu7=  0,  otherwise  U1>17=  1. 

[Pij]  =  [  STS  3  STS  4  STS  2]  therefore,  the  values  for  [Uu7]  for  STS  1  is  0  the  value 
for  STS  2  is  1. 


Untrained  Matrix 

During  the  unit’s  training  assessment  process  the  unit  is  evaluated  on  its 
performance  on  various  tasks.  The  unit  is  evaluated  as  trained  on  a  task  (T),  needing 
practice  (P),  or  untrained  on  the  task  (U)  (Training  Management,  1991).  The  MTP  task 
numbers  of  the  tasks  that  the  unit  is  untrained  in  are  placed  in  the  Untrained  Matrix  [Fy], 
This  information  is  required  to  evaluate  [U1>20]. 

Since  the  user  desires  to  consider  the  unit’s  assessment,  the  intersection  of  the 
Scenario  Task  Matrix,  [Ty],  and  the  Untrained  Matrix,  [Fy]  are  examined. 


~1352~ 

1352 

[Ty]  for  STS  1  = 

1642 

5429 

[Tij]  for  STS  2  = 

1142 
243 1_ 
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1476 

[Fy]  for  the  unit  =  1398  Therefore  the  values  for  [Uj  2o]  for  STS  1  is  0  the 
5429 

value  for  STS  2  is  1.  This  produces  the  following  revised  User  Preference  matrices: 

U  for  STS  1  =  [0  2  9  4  0  2  6  2  0  1  6  4  12  2  10  2  0  2  0  0] 

U  for  STS  2  =  [1  2  9  4  0  2  6  2  0  1  6  4  12  2  10  2  1  2  0  1] 

With  the  evaluation  of  the  of  the  final  element  of  the  User  Preference  Matrix  it  is 
necessary  to  evaluate  the  Difference  Matrix  with  respect  to  the  two  simulation  training 
scenarios. 

The  Scenario  Characteristic  Code  Matrices 

Table  43  summarizes  the  characteristics  of  the  two  training  scenarios.  The  values 
for  each  category  can  be  referenced  from  Tables  9  through  25  in  chapter  3.  By  definition 
the  values  of  [Su  j],  [S1;  17],  [S12o]  are  0. 
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TABLE  43  SCENARIO  CHARACTERISTIC  CODES 


Category 

SCC  for  STS  1 

SCC  for  STS  2 

Particular  task  to  be  trained 

0 

0 

Combat  Power  Ratio 

2 

2 

Enemy  Composition 

8 

9 

Enemy  Task  Organization 

4 

4 

Enemy  Equipment 

0 

0 

Enemy  Training  Level 

2 

2 

Enemy  Mission 

6 

6 

Exercise  Preparation  Time 

2 

1 

Mission  Planning  Time 

0 

0 

Terrain 

1 

1 

Weather 

6 

6 

Light  Data 

4 

4 

Friendly  Composition 

12 

12 

Friendly  Equipment 

2 

2 

Adjacent  Unit 

10 

10 

Supported  Observation  Devices 

2 

2 

Previous  Use  of  a  Scenario 

0 

0 

Level  of  Difficulty 

1 

2 

Simulator 

0 

0 

Unit  Assessment 

0 

0 

Table  43  yields  the  following  SCC  for  STS  1  and  STS  2: 
SCC  for  STS  1=  [0  2  8  4  0  2  0  6  2  0  1  6  4  12  2  10  2  0  1  0  0] 
SCC  for  STS  2=  [0  2  9  4  0  2  0  6  1  0  1  6  4  12  2  10  2  0  2  0  0] 
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The  Difference  Matrix 


The  Difference  matrix  is  formed  by  subtracting  the  SCC  matrix  from  the  UPC 
matrix.  The  Difference  Matrix  is  a  measure  of  the  conceptual  diastase  between  the 
simulation  training  scenario  and  the  ideal  simulation  training  scenario  represented  as  : 

Dweai  =  [0  00000000000000000  0] 

The  comparison  of  the  User  Preference  Matrices  and  the  Scenario  Characteristic 
Code  Matrices  yields  the  two  Difference  matrices: 

Dy  for  STS  1  =  [0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0] 

Dn  for  STS  2  =  [1  0  0  0  0  0  0  1  0  0  0  0  0  0  0  1  0  0  1] 

The  final  [U]  matrices  are  then  defined  as: 

[U]  for  STS  1  =  [1  1  0  1  1  1  1  1  1  1  1  1  1  1  1  1  0  1  1] 

[U]  for  STS  2  =  [0  1  0  1  1  1  1  0  1  1  1  1  1  1  1  0  0  1  0] 

Both  these  matrices  are  then  multiplied  by  the  Weight  matrix  [W], 


The  Rank  Matrix 

The  impact  of  the  weight  matrix  is  that  it  applies  the  relative  importance  of  each 
dimension  (factor)  as  determined  by  the  survey  analysis  to  the  selection  process. 
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Equation  3  results  in  the  Rank  Matrix.  This  number  is  a  measure  of  the  agreement 
between  the  users  preferences  and  the  candidate  simulation  training  scenarios. 

'  20 

2X^,1 

k= 1 

(equation  3) 

[Ru]for  STS  1  =  [12.5]  =13.0 
[Ru]for  STS  2  =  [10.4]  =  10.4 

The  larger  value  associated  with  STS  1  indicates  that  according  to  the  users  preferences, 
it  is  a  better  scenario  for  the  required  training. 


Automated  Sample  Tool 


The  automated  prototype  tool  is  written  using  Microsoft  Excel.  The  first  screen, 
Figure  2,  consists  of  the  same  User  Preference  matrix,  Task  matrix,  Previous  Use  matrix, 
Untrained  matrix,  and  Scenario  Task  matrices  as  in  the  above  example.  Due  to  the 
constraints  of  the  prototype  the  matrices  all  appear  in  the  form  of  [Element  I;  J.  For  all 
calculations  the  matrices  were  placed  in  their  true  form. 
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FIGURE  2:  AUTOMATED  TOOL  SCREEN  1 


Figure  2  also  shows  the  calculation  of  the  values  of  [Uj  J,  [Uj  17],  and  [U1;  20]  for 
both  STS  1  and  STS  2.  They  are  calculated  the  same  way  as  they  were  in  the  manual 


example. 


Figure  3  shows  the  revised  User  Preference  matrices  along  with  the  Scenario 
Characteristic  Codes  for  STS  1  and  STS  2.  The  Difference  matrices  are  also  calculated 
for  the  two  simulation  training  scenarios. 
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FIGURE  3:  AUTOMATED  TOOL  SCREEN  2 


Figure  4  shows  the  Final  User  Preference  matrices  for  the  two  simulation  training 
scenarios,  the  Weight  matrix  as  determined  from  the  analysis,  and  the  resulting  Rank 
matrices  for  the  two  simulation  training  scenarios.  The  larger  value  for  STS  1  indicates 
it  is  a  better  selection. 
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FIGURE  4:  AUTOMATED  TOOL  SCREEN  3 
Summary 

The  frequency  based  statistical  analysis  yielded  the  weights  needed  for  the 
implementation  strategy.  This  type  of  analysis  eliminated  any  problems  of  parameteric 
assumptions.  The  Preference-Characteristic  technique  proved  to  be  a  valid  technique  by 
which  to  differentiate  between  scenarios.  The  simplicity  of  the  method  lent  itself  easily 
to  manual  manipulation  and  a  prototype  automated  tool. 
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CHAPTER  7  FINDING,  LESSONS  LEARNED  AND  FUTURE  RESEARCH 

The  intent  of  this  effort  was  to  investigate  a  methodology  to  determine  factors  for 
selecting  training  scenarios  in  virtual  and  constructive  simulations.  The  research 
addressed  this  issue  in  four  areas: 

1.  How  is  simulation  training  used,  and  what  can  be  accomplished  to 
streamline  the  simulation  planning  process? 

2.  What  method  of  knowledge  acquisition  is  most  suitable  for  collecting 
information  from  a  widely  dispersed  non-homogeneous  population? 

3 .  What  means  of  analysis  are  the  most  appropriate  for  differentiating 
levels  of  significance  between  the  selection  factors? 

4.  How  can  this  information  be  applied  to  the  selection  of  simulation 
training  scenarios? 

Background 

The  effort  to  exploit  the  potential  of  simulation  for  improving  task  proficiency 
began  in  the  early  1970’s.  The  technology  of  the  time  limited  most  efforts  to 
improvements  in  live  simulation  training.  It  was  not  until  1973  that  the  use  of  computer 
tools  was  harnessed  to  the  effort.  Constructive  TES  was  the  first  computer  based 
simulation  training  device. 
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The  advancement  in  technology  and  the  successful  use  of  these  initial  training 
tools  lead  to  the  expansion  of  simulation  training  devices.  Along  with  the  explosion  of 
computer  based  simulations  was  an  order  of  magnitude  explosion  in  the  number  of 
training  scenarios.  The  number  of  scenarios  became  so  unwieldy  that  numerous 
governmental  review  boards  recommended  that  efforts  be  made  to  catalog  them  and 
place  them  in  libraries. 

To  control  this  problem,  the  Army’s  latest  major  simulation  program,  the 
Combined  Arms  Tactical  Trainer  (CATT),  developed  SATS-TREDS.  The  Standard 
Army  Training  Schedule-Training  Exercise  Development  System  searches  a  scenario 
database  and  presents  a  prioritized  list  of  appropriate  scenarios.  Members  of  the  Army 
simulation  and  training  community  have  also  emphasized  the  need  for  adding  additional 
types  of  searches.  This  type  of  automated  scenario  selection  streamlines  the  simulation 
training  planning  process. 

The  intended  user  group  is  very  diverse  and  geographically  spread  out.  The  need 
to  contact  a  large  number  of  members  from  each  group  drove  the  decision  to  use  a  survey 
to  gather  the  information.  The  expense  involved  in  conducting  face  to  face  interviews 
with  SME’s  or  utilizing  Nominal  Group  Techniques  makes  use  of  these  techniques 
prohibitive.  If  the  information  gathering  could  be  coupled  with  other  activities  the 
expense  is  reduced. 
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Findings 


The  survey  was  conducted  in  two  phases;  an  informal  pilot  phase  used  to  develop 
and  refine  the  survey  instrument  and  a  formal  pilot  where  members  of  the  targeted 
populations  were  solicited  for  their  responses.  The  informal  portion  ensured  that  the 
instrument’s  questions  addressed  the  appropriate  issues  and  could  be  easily  understood. 
During  this  phase  the  form  and  substance  of  the  questions  were  changed  to  ensure  that 
the  correct  information  could  be  gathered.  The  data  garnered  from  a  pre-sampling 
exercise  was  also  examined  using  the  intended  analysis  techniques  to  ensure  that  the  type 
of  analysis  was  appropriate. 

The  formal  pilot  study  gathered  data  from  members  of  the  target  population.  This 
formal  exercise  allowed  the  accumulation  of  representative  data  for  analysis  from  the 
target  population. 

Two  types  of  statistical  analysis  were  applied  to  the  data,  mean  based  and 
frequency  based.  The  mean  based  comparisons  required  a  great  deal  of  iterative 
calculations.  These  comparisons  revealed  statistical  differences  within  populations  and 
between  environments.  The  expansive  number  of  variables  (2380)  makes  this  type  of 
analysis  too  cumbersome  for  this  application.  Mean  based  comparisons  would  be 
appropriate  for  contexts  which  had  fewer  selection  factors  to  compare. 

The  frequency  based  analysis  was  conducted  using  chi-squared  tests  to  examine 
the  frequency  or  preference  levels.  This  analysis  also  revealed  differences  in  factor 
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levels,  but  failed  to  determine  differences  in  factors  between  the  two  environments. 
Though  this  type  of  analysis  may  not  be  as  precise,  it  easily  lends  itself  to  quantification. 

The  quantified  data  can  then  be  applied  through  a  mathematical  model  that  uses 
these  weights  to  differentiate  between  previously  equivalently  ranked  alternatives.  An 
automated  search  can  be  accomplished  by  quantifying  the  characteristics  of  the  scenarios 
and  the  users  preferences  against  the  same  criteria.  Simple  mathematics  and  matrix 
comparisons  and  calculations  can  then  be  used  to  quantify  the  level  of  agreement 
between  the  scenarios  and  the  users  preferences.  An  example  demonstrated  and  yielded  a 
correct  solution. 


Lessons  Learned 

The  most  important  lessons  learned  during  this  process  from  the  aspect  of  the 
research  are  related  to  the  distribution  and  analysis  of  the  instrument.  The  survey  process 
could  have  been  improved  by  taking  two  significant  steps.  The  most  important  step 
would  be  the  early  co-opting  of  an  organization  that  controlled  a  large  number  of  the 
members  of  the  target  population.  If  an  organization  could  have  been  brought  into  the 
process  at  the  survey  design  phase,  this  leverage  could  have  been  used  to  ensure  better 
participation  by  respondents  who  had  a  stake  in  the  analysis. 

Early  efforts  to  make  the  survey  an  on-line  document  or  electronic  form  would 
have  benefited  the  process.  Placement  of  the  survey  on  the  world  wide  web,  though 
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restricting  access  via  a  password,  would  have  improved  data  collection  and  organization. 
Distributing  the  survey  via  electronic  mail  would  have  also  improved  the  process.  This 
step  could  have  increased  distribution,  eased  data  categorization  and  analysis,  and 
increased  response  rate. 

The  final  lesson  concerns  the  types  of  statistical  analysis  used  to  determine  the 
weights.  Better  prior  analysis  of  the  informal  pilot  data  would  have  revealed  the 
exhaustive  nature  of  means  comparisons  for  such  a  large  sample  space.  This  effort 
would  have  lead  to  a  much  earlier  focus  on  the  frequency  based  statistical  analysis.  The 
frequency  based  analysis  used  in  this  research  is  the  basis  for  the  assignment  of  weights. 
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Further  Research 


Efforts  should  be  made  to  implement  this  methodology  in  refining  simulation 
scenario  searches.  Though  the  research  was  conducted  within  a  military  training 
environment,  this  methodology  can  be  applied  in  any  domain.  The  process  of  eliciting 
information  from  the  user  population,  conducting  discriminating  analysis  to  determine 
levels  of  importance,  and  designing  tools  to  automate  the  application  of  the  knowledge 
can  be  beneficial  in  many  situations. 

Work  can  be  continued  on  the  studied  problem  as  well.  Based  on  the  results  of 
the  pilot  study,  an  effort  should  be  made  to  gain  wider  distribution  of  the  survey  as  an 
official  Army  survey.  This  could  be  accomplished  in  many  ways.  The  Army  Survey 
Office  should  be  utilized  to  distribute  the  survey  over  a  much  wider  range.  This  would 
require  a  formal  proposal  and  funding  for  distribution,  collection,  and  some  automation 
of  analysis.  The  survey  could  also  be  placed  on  line.  Links  could  be  placed  on  official 
Army  web  sites  and  the  survey  could  be  converted  to  an  interactive  document.  The 
conversion  to  an  interactive  format  would  ease  the  problems  of  data  tabulation  and 
analysis.  A  great  deal  of  effort  would  be  required  to  coordinate  for  permission  to 
establish  links  from  other  Army  sites  and  to  establish  some  kind  of  screening  process  to 
prevent  unqualified  respondents  from  contaminating  the  database. 

Once  data  has  been  collected  the  selection  methodology  should  be  automated. 
The  automated  system  could  be  integrated  into  the  SATS,  SATS-TREDS  or  any  other 
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planning  tool  and  other  scenario  library  devices.  Resource  Consultants  Incorporated  is 
currently  working  on  the  Close  Combat  Tactical  Trainer  (CCTT)  scenario  library.  In 
their  design  they  have  included  a  METT-T  factor  selection  screen.  There  is  currently  no 
code  or  algorithms  functioning  based  on  the  screen,  but  the  interface  could  be  easily 
modified  to  support  scenario  searches  based  on  the  methodology. 

Further  research  could  also  be  conducted  into  use  of  other  scenario  screening 
factors.  The  Battlefield  Operating  Systems,  Combat  Imperatives,  Principles  of  War ,  and 
Combat  Instruction  Sets  are  also  good  candidates  for  selection  factors.  This  same 
methodology  could  be  applied  to  determine  their  importance. 

The  methodology  is  also  applicable  in  any  industry  that  utilizes  simulation  based 
training.  The  tool  designer  must  establish  valid  candidate  selection  factors,  collect  and 
analyze  data  to  determine  the  appropriate  weights  for  those  factors,  categorize  the 
scenario  database  and  user  preferences  using  those  factors,  and  apply  the  Preference- 
Characteristic  Matrix  Technique. 
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Appendix  A 
Glossary 
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ANOVA 

Analysis  of  Variance 

AGTS 

Advanced  Gunnery  Training  Simulator 

ARI 

Army  Research  Institute 

ARTEP 

Army  Training  and  Evaluation  Plan 

BBS 

Brigade  Battle  Simulation 

BDE 

Brigade 

BFDT 

Board  for  Dynamic  Training 

BN 

Battalion 

CBS 

Corps  Battle  Simulation 

CATT 

Combined  Arms  Tactical  Trainer 

CCTT 

Close  Combat  Tactical  Trainer 

CTC 

Combat  Training  Center 

cs  &  css 

Combat  Support  and  Combat  Service  Support 

df 

Degrees  of  freedom 

DIS 

Distributed  Interactive  Simulation 

GUARDFIST I 

Call  for  Fire  Trainer 

HLA 

High  Level  Architecture 

HQDA 

Headquarters  Department  of  the  Army 

Janus 

A  constructive  simulation 

JCM 

Joint  Combat  Model 

LAN 

Local  Area  Net 
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METL 

METT-T 

MTP 

MILES 

MTS 

NGT 

NTC 

NTSA 

OPFOR 

PCC 

PDU 

PM 

REALTRAIN 

REFORGER 

SAF 

SATS-TREDS 

SCC 

SCOPES 


Mission  Essential  Task  List 

Mission,  enemy,  terrain,  troops,  and  time 

Mission  Training  Plan 

Multiple  Integrated  Laser  Engagement  System 
Moving  Target  Simulator 
Nominal  Group  Technique 
National  Training  Center  at  FT.  Irwin 
National  Training  Systems  Association 
Opposing  Forces 
Preference-Characteristic  Code 
Protocol  Data  Unit 
Project  Manager 

Early  vehicle/crew  live  simulation  engagement 
trainer 

Re-enforce  Germany  Training  Exercise 

Semi-automated  Forces 

Standard  Army  Training  System-Training 

Exercise  Development  System 

Scenario  Characteristic  Code 

Early  small  unit  live  simulation  engagement 

trainer 
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SIMNET 

SME 

STS 

TADSS 


TPS  Codes 

TES 

TWGSS 

UCOFT 

UPC 


Simulation  Networking  Trainer 
Subject  Matter  Expert 
Simulation  Training  Scenario 
Training  Aides,  Devices,  Simulators  and 
Systems 

Task  Performance  Support 

Tactical  Engagement  System 

Tank  Weapons  Gunnery  Simulation  System 

Unit  Conduct  of  Fire  Trainer 

User  Preference  Code 
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Appendix  B 


User  Preference  Code  and  Scenario  Characteristic  Code  Elements 
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TABLE  9  U12  Su  VALUES 


User  Preference 

Value  of  U1>2 

CPR>  doctrinal  level 

3 

CPR  =  doctrinal  level 

2 

CPR<  doctrinal  level 

1 

No  Preference 

0 

TABLE  10  U1(3  S1>3:  VALUES 


User  Preference 

KgHTffllTilfli 

Motorized  Rifle  Division 

16 

Motorized  Rifle  Regiment 

15 

Motorized  Rifle  Battalion 

14 

Motorized  Rifle  Company 

13 

Tank  Division 

12 

Tank  Regiment 

11 

Tank  Battalion 

10 

Tank  Company 

9 

Airborne  Division 

8 

Air  Assault  Brigade 

7 

Air  Mobile  Assault  Brigade 

6 

Infantry  Pure 

5 

Armor  Pure 

4 

Aviation  Pure 

3 

Air  Defense  Pure 

2 

Field  Artillery  Pure 

1 

No  Preference 

0 

TABLE  11  U1)4  S1)4  VALUES 


j  User  Preference 

Main  Body 

6 

Forward  Security  Element 

5 

Advanced  Guard 

4 

Mobile  Obstacle  Detachment 

3 

Command  Post 

2 

Reconnaissance  Element 

1 

No  Preference 

0 
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TABLE  12  U1)SS1)S  VALUES 


|  User  Preference 

Value  of  U1S 

Tank 

Infantry 

Tactical  Air 

T-80 

BMP-3 

Holkum 

5 

1  Any  one  of  the  above  but  not  all  three 

4 

T-72 

BMP-2 

Hind  series 

3 

Any  one  of  the  above  but  not  all  three 

2 

Other 

1 

No  Preference 

0 

TABLE  13  U1;6  Sh6  VALUES 


User  Preference 

Enhanced  Probability  of  Hit,  Kill,  or  Detection 

3 

Standard  Probability  of  Hit,  Kill,  or  Detection 

2 

Depressed  Probability  of  Hit,  Kill,  or  Detection 

1 

No  Preference 

0 

TABLE  14  U1>7  S1>7  VALUES 


User  Preference 

Offensive  Operations 

Attack 

6 

Meeting  Engagement 

5 

Pursuit 

4  ; 

Defensive  Operations 

Hasty  Defense 

3 

Prepared  Defense 

2 

Withdrawal 

1 

No  Preference 

0 

112 


TABLE  15  U1>8  S1>8  VALUES 


User  Preference 

Value  of  U1>8 

Complete  Training  Support  Package 

2 

Partial  Training  Support  Package 

1 

No  Preference 

0 

TABLE  16  U1>9  S1>9  VALUES 


User  Preference 

Value  of  U19 

Unit  Required  to  Execute  Supplied  Orders 

2 

Unit  must  Execute  the  Mission  Planning  Process 

1 

No  Preference 

0 

TABLE  17  Uwo 

S140  VALUES 

User  Preference 

Value  of  U140 

Mountainous 

6 

Swamp 

5 

Jungle 

4 

Wooded 

3 

Rolling  Hills 

2 

Desert 

1 

No  Preference 

0 

TABLE  18  UW1 

SW1  VALUES 

User  Preference 

Value  of  U1>n 

Clear 

6 

Light  Rain 

5 

Rain 

4 

Snow 

3 

Fog 

2 

Sleet/Icy 

1 

No  Preference 

0 
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TABLE  19  U142  SW2  VALUES 
User  Preference  Value  of  U1>12 


Daylight  5 


Dusk 


Night  (Moon)  _ 3 


Night  (No  moon)  | _ 2 


Dawn  1 


No  Preference  0 


TABLE  20  UW3  SU3  VALUES 
User  Preference  Value  of  U143 


Brigade  _ 14 


Task  Force  13 


Battalion  12 


Squadron _ 11 


Company  _ 10 


Battery  9 


Troop  _ 8 


Platoon  7 


Section  6 


Detachment  5 


Squad 


Crew  3 


Team  2 


No  Preference  0 


TABLE  21  UU4  SW4  VALUES 


User  Preference 

Value  of  U114 

Current  Table  of  Organization  and  Equipment 

2 

Other 

1 

No  Preference 

0 
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TABLE  22  U1;15  S1;15  VALUES 


User  Preference 

Value  of  Uj5i5 

United  States  Army 

10 

United  States  Marines 

9 

British 

8 

German 

7 

Kuwaiti 

6 

Saudi  Arabian 

5 

French 

4 

Canadian 

3 

Korean 

2 

Other 

1 

No  Preference 

0 

TABLE  23  UU6  Sw6  VALUES 


User  Preference 

Value  of  U116 

All  Imaging  Equipment 

4 

Radar,  Thermal,  and  Optical 

3 

Thermal  and  Optical 

2 

Optical 

1 

No  Preference 

0 

TABLE  24  UU8  SM8  VALUES 


User  Preference 

Value  of  U1>18 

More  Difficult 

3 

Standard 

2 

Less  Difficult 

1 

No  Preference 

0 
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TABLE  25  U1>19  SW9  VALUES 


User  Preference 

Value  of  U119 

AGTS 

14 

BBS 

13 

CBS 

12 

CCTT 

11 

Driver  Trainer 

10 

JANUS 

9 

JCM 

8 

MTS 

7 

Pilot  Trainer 

6 

SIMNET 

5 

GUARDFIST  I 

4 

TSFO 

3 

UCCATS 

2 

UCOFT 

1 

No  Preference 

0 
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Appendix  C 


Survey  Questions’  Intent 
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This  section  is  designed  to  collect  the  demographic  data  that  will  determine  if  the 
respondent  is  qualified  to  respond  to  the  questionnaire.  It  can  also  be  used  to 
differentiate  the  different  sub-groups  of  the  respondent  population. 

Please  answer  the  following  questions.  All  information  will  be  kept  in  strict  confidence.  Only  aggregate 
data  will  be  reported. 


A.  General 

Name  (optional): 

Branch:  Rank:_ 

Unit  Current  Duty  Position:_ 

Last  Level  of  Command: 

Military  Education  Level: 

Age:  Time  in  Service: 

Time  in  Unit _  Previous  duty  positions  over  the  last  five  years: 


B.  Training  and  Simulation 

For  the  following  questions,  circle  the  most  appropriate  response. 

1 .  If  you  related  your  unit  training  experience  in  terms  of  years,  where  would  you  place  yourself  on  the 

chart  below  ?  _  _ _ _ 

less  than  5  6-10  11-15  16-21  over  21 

years  years  years  years  years 

2.  In  the  last  year,  how  much  total  time  have  you/your  staff  spent  planning  simulation  based  training? 


less  than  10  11-20  21  -30  31  -40  over41 

hours  hours  hours  hours  hours 


3.  In  the  last  year,  how  much  time  has  your  unit  spent  executing  simulation  based  training? 


less  than 
1  day 


1-5 

days 


6-10 

days 


10-15 

days 


over  16 
days 
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4.  Of  your  simulator  training  time,  how  much  is  in  a  virtual  environment  (SIMNET,  UCOFT,  Flight 
Trainer,  etc.)? 


less  than  1-5  6-10  10-15  over  16 

1  day  days  days  days  days 

5.  Of  your  simulator  training  time,  how  much  is  in  a  constructive  environment  (JANUS,  BBS,  CSSTSS, 
CBS,  etc.)? 


1 

less  than 

i 

1-5 

i 

6-1 

10 

10-15 

over  16 

1  day 

days 

days 

days 

days 

6.  How  effective  do  you  believe  simulation  based  training  is  in  preparing  your  unit  for  war? 


Not  effective  Somewhat  ineffective  Effective  Highly  effective  Very  highly  effective 

0-20%  21-40%  41-60%  61-80%  81-100% 


Question  7  will  elicit  the  simulators  with  which  the  subject  has  trained.  The  subjects 
estimation  of  effectiveness  may  also  indicate  biases.  All  acronyms  are  included  in  the 

glossary. 


7.  Which  simulations  or  simulator  devices  have  you  or  your  unit  trained  with,  and  how  would  you 
evaluate  their  effectiveness?  (Please  check) 


Simulator 

I  or  my  unit  have 
trained  with  this 
simulator 

Not  effective 
0-20% 

Somewhat 

ineffective 

21-40% 

Effective 

41-60% 

Highly 

effective 

61-80% 

Very  Highly 
Effective 
81-100% 

AGTS 

BBS 

CBS 

CCTT 

Driver  Trainer 

JANUS 

JCM 

MTS 

Pilot  Trainer 

SIMNET 

GUARDFIST  I 

TSFO 
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ADDITIONAL  COMMENTS: 


All  of  these  questions  refer  to  factors  that  may  or  may  not  influence  your  selection  of  training 
scenarios  to  be  utilized  in  constructive  and  virtual  simulations.  Please  answer  them  by  placing  a 
numerical  value  corresponding  to  your  level  of  belief  that  you  would  use  each  factor  in  selecting  a 
training  scenario.  You  may  use  any  value  between  0  and  100.  The  scale  below  is  meant  only  as  a 
guide.  You  are  to  give  answers  for  both  virtual  and  constructive  environments.  It  may  help  to 
imagine  that  you  are  planning  virtual  (SIMNET,  UCOFT,  PGT,  CCTT,  Pilot  trainer,  etc.)  and 
constructive  (JANUS,  BBS,  CBS,  etc.)  training  exercises.  What  are  the  chances  you  consider  these 
factors? 

The  scale  is  designed  to  help  the  respondent  to  focus  his  response.  The  use  of  the 
numerical  and  verbal  scale  eliminates  ambiguity. _ 


Very  Small  Chance  Small  Chance  Could  Effect  High  Chance  Very  High  Chance 

0-20%  21-40%  41-60%  61-80%  81-100% 

1.  How  much  does  the  type  of  terrain  (desert,  rolling  hills,  restrictive,  etc.)  affect  your  scenario 
selection? 

INTENT:  To  determine  the  value  the  battle  space  land form  has  on  the 

selection 

EFFECT:  If  this  factor  is  not  important  then  scenarios  do  not  need  to  be 

created  on  terrain  bases  representing  the  different  geographic 
operational  areas.  This  will  reduce  the  number  of  scenarios 
significantly  and  eliminate  the  need  to  produce  variants  for 
different  theaters. 


If  this  factor  is  important  the  scenario  library  must  contain  a 
variant  of  every  mission  on  each  terrain  base. 

Keyed:  Terrain  is  currently  tracked  in  the  training  support  package. 


2.  How  much  does  one  particular  task  to  be  trained  (individual  through  collective)  affect  your 
scenario  selection? 

INTENT:  To  determine  if  a  particular  task  can  effect  the  scenario 

selection. 

EFFECT:  If  this  factor  is  not  important  then  the  current  task  list  comparison  used 

in  SA  TS-TREDS  is  acceptable.  The  most  matches  between  the  desired 
task  list  and  the  training  scenario ’s  supported  task  list  is  desirable. 

If  this  factor  is  important  the  search  criteria  must  be  altered  so  that  a 
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training  scenario  must  include  the  “key  ”  task  to  be  acceptable. 

Keyed:  Supported  training  tasks  are  currently  tracked  in  the  training  support 

package. 


3.  How  much  does  the  simulator  type  (JANUS,  CBS,  BBS,  S1MNET,  CCTT  etc.)  affect  your 
scenario  selection? 

INTENT:  To  determine  the  value  of  limiting  the  search  to  a  particular 

simulator. 

EFFECT:  If  this  factor  is  not  important  then  all  of  the  scenarios  must  be 

examined.  This  would,  be  useful  for  long  term  planning  before 
resources  constraints  are  considered.  This  would  allow  the  trainer  to 
select  the  best  scenario  available  to  train  the  desired  tasks  and  then  look 
to  resource  that  plan. 

If  this  factor  is  important  the  training  planner  must  consider  resource 
constraints  prior  to  refining  his  search. 

Keyed:  Each  simulation  type  is  currently  tracked  in  the  training  support 

package. 

4.  How  much  does  the  friendly  unit’s  composition  (pure  or  combined  arms)  affect  your  scenario 
selection? 

INTENT:  To  determine  the  value  the  friendly  units  make  up  has  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  the  library  will  be  searched  without 

regard  to  friendly  force  structure. 

If  this  factor  is  important  the  search  will  have  to  consider  the  types  of 
units  and  the  types  of  vehicle  crew  configurations  that  the  scenario  is 
designed,  to  support. 

Keyed:  Friendly  unit ’s  composition  is  currently  tracked  in  the  training  support 

package. 

5.  How  much  does  the  enemy  unit’s  composition/capabilities  (motorized,  light,  armored,  attack 
aviation,  etc.)  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  of  enemy ’s  characteristics  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  very  generic  enemy  force  templates 

can  be  created.  Such  templates  could  be  built  on  battalion,  regiment, 
division  and  corps  frameworks  and  be  associated  with  any  terrain  and 
task  list. 
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If  this  factor  is  important  the  scenario  library  must  contain  a  variant  of 
every  mission  on  each  terrain  base. 

Keyed:  Enemy  unit ’s  composition/capabilities  are  currently  tracked  in  the 

training  support  package. 


6.  How  much  does  the  enemy/friendly  combat  power  ratio  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  combat  power  ratios  have  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  generic  enemy  force  templates  may  be 

inserted  in  any  scenario  in  the  library. 

If  this  factor  is  important  the  doctrinally  correct  combat  power  ratios 
should  be  referenced  in  the  scenario  selection. 

Keyed:  Combat  power  ratio  is  not  currently  tracked  in  the  training  support 

package. 


7.  How  much  does  the  enemy’s  task  organization  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  enemy ’s  task  organization  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  enemy  combat  elements  such  as  the 

regimental  recon,  advanced  guard,  or  a  motorized  rifle  regiment  can  be 
standardized  with  little  need  to  adjust  later. 

If  this  factor  is  important  the  scenario  library  must  contain  variations  of 
enemy  force  structure. 

Keyed:  Enemy  force  structure  is  not  currently  tracked  in  this  form. 


8.  How  much  does  enemy  equipment  (T-62  vs.  T-72  or  Hind-D  vs.  Holkum)  affect  your  scenario 
selection? 

INTENT:  To  determine  the  value  enemy  equipment  types  have  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  scenarios  can  be  created  with 

“typical”  enemy  equipment.  This  will  reduce  the  size  of  the  data  base 
required  to  support  the  simulations. 

If  this  factor  is  important  the  scenario  library  must  track  scenario 
variants  with  different  enemy  equipment.  .  This  causes  a  need  for  a 
larger  data  base  to  support  the  simulation  and  the  ability  to  add  or 
modify  equipment  in  the  scenario. 

Keyed:  Enemy  equipment  is  currently  tracked  in  the  training  support  package. 


Keyed:  Enemy  equipment  is  currently  tracked  in  the  training  support  package. 


9.  How  much  does  friendly  equipment  (M60  vs.  Ml  A2  or  Cobra  vs.  Apache  Longbow)  affect  your 
scenario  selection? 

INTENT:  To  determine  the  value  friendly  unit  equipment  has  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  all  of  the  scenarios  can  be  built  with 

“typical”  friendly  equipment.  This  will  reduce  the  size  of  the  data  base 
associated  with  the  simulations. 


If  this  factor  is  important  the  scenario  library  must  be  built  to 
accommodate  different  friendly  equipment  types.  This  causes  a  need for 
a  larger  data  base  to  support  the  simulation  and  the  ability  to  add  or 
modify  equipment  in  the  scenario. 

Keyed:  Friendly  equipment  is  currently  tracked  in  the  training  support  package. 


10.  How  much  does  adjacent  friendly  unit  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache 
Longbow)  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  adjacent  friendly  unit  equipment  types  have  on 

the  selection.  This  will  support  the  simulation  of  multinational  forces 
fighting  in  adjacent  areas  of  the  battle  field. 

EFFECT:  If  this  factor  is  not  important  all  of  the  scenarios  can  be  built  with 

“typical”  friendly  equipment.  This  will  reduce  the  size  of  the  data  base 
associated  with  the  simulations 


If  this  factor  is  important  the  scenario  library  must  be  built  to 
accommodate  different  equipment  types.  This  causes  a  need  for  a  larger 
data  base  to  support  the  simulation  and  the  ability  to  add  or  modify 
equipment  in  the  scenario.  This  will  also  drive  the  need  to  depict  all 
equipment  types  in  a  ’’friendly  ”  and  ” enemy  ”  mode. 

Keyed:  Adjacent  unit  equipment  is  not  clearly  defined  in  the  training  support 

package. 


1 1 .  How  much  does  the  enemy’s  training  level  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  enemy ’s  training  level  has  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  standard  values  for  enemy  training 

indicators  such  as  the  time  to  acquire  a  target  and  the  rate  of fire  of  a 
weapon  system  can  be  incorporated  in  all  of  the  scenarios. 
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If  this  factor  is  important  these  training  indicators  must  be  coded  in 
such  a  way  that  they  can  be  changed  for  particular  units  in  a  given 
scenario  (perhaps  for  recently  reconstituted  units).  They  may  be  able  to 
be  correlated  to  a  level  of  difficulty  associated  with  the  scenario. 

Keyed:  The  enemy ’s  training  level  is  not  currently  tracked  in  the  training 

support  package. 


12.  How  much  does  the  enemy’s  mission  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  enemy ’s  mission  has  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  it  will  reduce  the  number  of  scenarios  that 

need  to  be  maintained  in  the  library. 

If  this  factor  is  important  the  number  of  scenarios  will  increase 
substantially.  The  library >  will  need  to  contain  all  combinations  of 
friendly  unit  missions  versus  enemy  unit  missions  for  all  other  factors 
that  are  significant. 

Keyed:  The  enemy ’s  mission  is  currently  tracked  in  the  training  support 

package. 


13.  How  much  does  weather  (fog,  rain,  or  haze)  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  weather  has  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  weather  effects  can  be  either  removed 

from  the  scenarios  or  added  as  a  random  variable  dependent  on  the 
terrain  form  being  utilized. 

If  this  factor  is  important  the  scenario  library  must  contain  scenario 
variations  that  duplicate  the  weather  patterns  associated  with  the 
terrain  forms.  It  would  also  become  necessary  to  enable  the  trainer  to 
modify  weather  conditions  on  demand  within  the  scenario. 

Keyed:  Weather  is  not  currently  tracked  in  the  training  support  package. 


14.  How  much  does  previous  use  of  a  scenario  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  previous  use  of  a  scenario  have  on  the  selection. 

This  will  support  the  desire  of  a  commander  to  expose  his  soldiers  to 
new  conditions  under  which  to  practice  a  task.  Or  the  ability  to  re-train 
a  soldier/unit  on  an  important  scenario  with  which  they  have  had 


125 


difficulty  in  the  past. 

EFFECT:  If  this  factor  is  not  important  no  preference  will  be  given  to  a  scenario 

based  on  its  prior  use  or  lack  of  use. 

If  this  factor  is  important  the  simulation  selection  tool  must  record 
previous  use  of  a  scenario  at  the  soldier  or  unit  level  as  well  as  the 
trainer  preference  for  re-use. 

Keyed:  The  previous  use  of  a  scenario  is  not  currently  tracked  in  the  training 

support  package. 


15.  How  does  the  level  of  difficulty  of  the  exercise  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  level  of  difficulty  of  the  exercise  has  on  the 

selection.  This  factor  would  allow  commanders  to  execute  crawl,  walk, 
run  training  with  their  units  more  easily.  It  would  also  allow 
soldiers/units  to  retrain  on  tasks  they  have  had  difficulty  executing  at  an 
easier  level. 

EFFECT:  If  this  factor  is  not  important  no  changes  need  to  be  made  to  the  current 

selection  technique. 

If  this  factor  is  important  an  algorithm  for  determining  the  level  of 
difficulty  for  a  simulation  exercise  must  be  developed.  The  algorithm 
must  consider  as  a  minimum  simulator  and  mission.  This  information 
would  then  have  to  be  coded for  the  search. 

Keyed:  The  level  of  difficulty  of  the  exercise  is  not  currently  tracked  in  the 

training  support  package. 


16.  How  much  does  unit  past  performance  assessment  (T's.  P’s,  and  U’s)  affect  you  scenario 
selection? 

INTENT:  To  determine  the  value  unit  past  performance  assessment  has  on  the 

selection. 

EFFECT:  If  this  factor  is  not  important  there  is  no  need  to  reference  training 

assessment  during  selection. 

If  this  factor  is  important  it  would  have  to  be  associated  with  a  priority 
list.  A  commander  would  need  to  prioritize  mission  essential  task  list 
(METL)  elements  according  to  their  assessment.  This  would  affect  the 
selection  of  the  scenario  both  in  terms  of  supported  tasks  and  level  of 
difficulty.  (You  would  not  want  to  have  an  untrained  unit  attempt  to 
execute  a  task  under  the  most  difficult  conditions.) 
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Keyed:  Unit  past  performance  assessment  is  currently  tracked  by  the  SA  TS- 

TREDS  tool. 


17.  How  much  do  light  conditions  (daylight,  night,  %  illumination,  etc.)  affect  your  scenario 
selection? 

INTENT:  To  determine  the  value  light  conditions  have  on  the  selection. 

EFFECT:  If  this  factor  is  not  important  then  each  scenario  can  be  established  with 

a  single  time(local  noon  or  midnight).  This  reduces  the  computational 
load  on  the  simulator  with  the  factors  affected  by  lighting(shadow, 
detection,  etc.)  Only  scenarios  that  included  tasks  that  required 
darkness  would  be  constructed  to  include  night. 

If  this  factor  is  important  each  scenario  would  have  to  be  built  with  a 
scenario  time  clock.  This  would  require  the  simulator  to  make 
calculations  based  on  position  of  the  sun  and  the  moon.  This  adds 
computational  a  load  to  the  system.  The  clock  would  need  to  be 
adjustable  to  be  able  to  start  the  scenario  at  any  scenario  local  time. 

Keyed:  Light  conditions  are  not  currently  tracked  in  the  training  support 

package  outside  of  tasks  such  as  “Conduct  night  attack.  ” 


1 8.  How  much  does  the  capability  to  support  observation  devices  (FLIR,  Thermal,  NVG,  Optic,  etc.) 
and  their  degradation  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  capability  to  support  observation  devices  has 

on  the  selection. 

EFFECT:  If  this  factor  is  not  important  the  scenarios  can  be  used  by  nits  that  have 

all  versions  of  the  same  equipment.  This  reduces  the  number  of 
scenarios  and  their  configurations. 

If  this  factor  is  important  scenarios  have  to  be  built  maintained 
specifically  to  support  each  improvement  in  sensing. 

Keyed:  The  capability  to  support  observation  devices  is  currently  tracked  by  the 

type  of  simulator. 


19.  How  much  does  the  amount  of  exercise  preparation  required  (placement  of  forces  on  the  battle 
field,  coordination  of  simulation  facility  use,  and  other  administrative  tasks  associated  with  the 
training)  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  amount  of  exercise  preparation  required  has 

on  the  selection.  This  affects  the  quality  of  the  training  support  package 


needed  to  accompany  each  scenario. 

EFFECT:  If  this  factor  is  not  important  then  each  scenario/training  support 

package  must  be  built  to  a  predetermined  level  of  completeness.  The 
level  of  completeness  addresses  items  such  as  unit  placement  on  the  land 
forms,  unit  scheduling,  and  training  schedule  planning. 

If  this  factor  is  important  then  all  scenarios  must  still  be  built  to  a  set 
standard,  but  certain  scenario ’s  should  have  placement  and 
coordination  data  that  can  be  used  to  support  the  training.  This  may  be 
important  for  reserve  and  national  guard  units  that  may  not  have  the 
resources  and  time  to  do  a  complete  training  cycle. 

Keyed:  The  amount  of  exercise  preparation  required  is  not  currently  tracked  in 

the  training  support  package. 


21.  How  much  does  the  amount  of  mission  planning  required  (troop  leading  procedures,  course  of 
action  development,  OPORD  preparation)  affect  your  scenario  selection? 

INTENT:  To  determine  the  value  the  amount  of  mission  planning  required  has  on 

the  selection. 

EFFECT:  If  this  factor  is  not  important  the  all  scenarios  will  be  built  to  allow  the 

trained  unit  to  conduct  a  complete  planning  cycle.  The  TSP  will  include 
a  complete  higher  headquarters’  operations  order  (OPORD). 

If  this  factor  is  important  then  some  scenario  TSP ’s  will  include 
complete  orders  and  execution  matrixes  for  the  higher  unit,  the  trained 
unit  and  any  other  elements  taking  part  in  the  training.  This  will  allow 
some  units  who  lack  planning  time  to  train  on  operational  execution. 

Keyed:  The  amount  of  mission  planning  required  is  currently  tracked  in  the 

training  support  package. 


22.  Please  list  below  any  other  factors  that  affect  your  scenario  selection? 

FACTOR: _ 

INTENT:  To  determine  if  other  potentially  important  factors  affecting  scenario 

selection  have  been  left  out  of  the  survey. 

EFFECT:  If  factors  are  repeatedly  noted  on  the  form  or  seem  particularly 

applicable  to  the  process  they  will  be  included  on  the  survey  used  in  the 
definitive  study. 

Keyed:  Not  applicable 
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Additional  Comments: 
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Date _ 

Please  answer  the  following  questions  by  circling  your  response.  All  information  will  be  kept  in  strict 
confidence.  Only  aggregate  data  will  be  reported. 


A.  General 

Name  (optional): _ 

Current  Duty  Position: _ 

Military  Education  Level: 
Time  in  Unit _ 


_  Branch: _  Rank: 

_  Last  Level  of  Command: _ 

_  Age: _  Time  in  Service: _ 

Previous  duty  positions  over  the  last  five  years: _ 


B.  Training  and  Simulation 


1.  If  you  related  your  unit  training  experience  in  terms  of  years  where  would  you  place  yourself  on  the 
chart  below  ? 


less  than  5  6-10  11-15  16-21  over  21 

years  years  years  years  years 

2.  In  the  last  year,  how  much  time  have  you/your  staff  spent  planning  simulation  based  training? 


less  than  10  11-20  21  -30  31  -40  over  41 

hours  hours  hours  hours  hours 

3.  In  the  last  year,  how  much  time  has  your  unit  spent  in  simulator  based  training? 


less  than 
1  day 


1-5 

days 


6-10 

days 


10-15 

days 


over  16 
days 
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4.  Of  your  simulator  training  time,  how  much  is  in  a  virtual  environment  (SIMNET,  UCOFT,  Flight 
Trainer  etc.)? 


less  than 
1  day 


1-5 

days 


6-10 

days 


10-15 

days 


over  16 
days 


5.  Of  your  simulator  training  time,  how  much  is  in  a  constructive  environment  (JANUS,  BBS,  CSSTSS, 
CBS,  WARSIM,  etc.)? 


less  than 

i 

1-5 

i 

6-1 

[0 

10-15 

l 

over  16 

1  day 

days 

days 

days 

days 

6.  Does  your  unit  use  simulation  as  a  capstone  training  event? 

Yes _  No 


7.  Which  simulations  or  simulator  devices  have  you,  personally,  trained  with,  and  how  would  you 
evaluate  their  effectiveness?  (Please  check.) 


Simulator 

I  have  personally 
trained  with  this 
simulator 

Not  effective 
0-20% 

Somewhat 

ineffective 

21-40% 

Effective 

41-60% 

Highly 

effective 

61-80% 

Very  Highly 
Effective 
81-100% 

AGTS 

BBS 

CBS 

CCTT 

Driver  Trainer 

JANUS 

JCM 

MTS 

Pilot  Trainer 

SIMNET 

STOW(E) 

TSFO 

UCCATS 

UCOFT 

! 

OTHERS: 
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ADDITIONAL  COMMENTS: 


The  following  questionnaires  are  designed  to  determine  what  factors  effect  a  commander’s  or 
operations  officer’s  selection  of  a  simulation  training  scenario.  The  intent  is  to  determine  which 
factors  or  properties  are  the  most  important  for  virtual  simulations,  such  as  CCTT,  and  which  are 
most  important  for  constructive  simulations,  such  as  JANUS  or  CBS. 

The  factors  that  are  determined  to  be  the  most  important  will  be  converted  to  an  expert 
system  that  will  be  incorporated  into  SATS-TREDS.  As  part  of  SATS-TREDS,  the  commander  will 
be  able  to  tailor  them  to  aid  in  selecting  the  appropriate  training  scenario  from  the  army’s  simulation 
library.  The  selection  process  will  be  interactive;  the  commander  will  be  presented  with  5  or  6 
scenarios  that  closely  meet  his  needs.  Each  choice’s  strengths  will  be  highlighted  to  help  the 
commander  make  his  decision. 

The  separate  questionnaires  allow  the  commander  to  emphasize  factors  that  he  thinks  can  be 
better  trained  in  each  environment. 
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Date 


Name  (optional) _  Branch _  Rank 

Current  Duty  Position _ 

Previous  duty  positions  over  the  last  five  years _ 


All  of  these  questions  refer  to  factors  that  may  or  may  not  influence  your  selection  of  training 
scenarios  to  he  utilized  in  constructive  simulations.  Please  answer  them  by  placing  a  numerical  value 
corresponding  to  your  level  of  belief  that  you  would  use  each  factor  in  selecting  a  training  scenario. 
You  may  use  any  value  between  0  and  100.  The  scale  below  is  meant  only  as  a  guide. 


Very  Small  Chance  Small  Chance  Could  Effect  High  Chance  Very  High  Chance 

0-20%  21-40%  41-60%  61-80%  81-100% 

1.  How  much  does  the  type  of  terrain  (dessert,  rolling  hills,  restrictive  etc.)  affect  your  selection? 

Value 


2.  How  much  does  one  particular  task  affect  your  selection? 

Value 


3.  How  much  does  the  simulator  selection  (JANUS,  CBS,  BBS  etc.)  affect  your  selection? 

Value  P 


4.  How  much  does  the  friendly  unit’s  composition  (pure  or  combined  arms)  affect  your  selection? 

Value 


5.  How  much  does  the  enemy  unit’s  composition/capabilities  (motorized,  light,  armored,  attack 
aviation  etc.)  affect  your  selection? 

Value 
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6.  How  much  does  the  enemy/friendly  ratio  affect  your  selection? 

Value 


7.  How  much  does  enemy  equipment  (T-62  vs.  T-72  or  Hind  D  vs.  Holkum)  affect  your  selection? 

Value 


8.  How  much  does  friendly  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache  Longbow)  affect  your 
selection? 

Value 

9.  How  much  does  friendly  bordering  unit  equipment  (M60  vs.  Ml  A2  or  Cobra  vs.  Apache 
Longbow)  affect  your  selection? 

Value 


10.  How  much  does  enemy’s  training  level  affect  your  selection? 

Value 


11.  How  much  does  enemy’s  strength  level  affect  your  selection? 

Value 


12.  How  much  does  the  enemy’s  mission  affect  your  selection? 

Value 


13.  How  much  does  weather  affect  your  selection? 


Value 


14.  How  much  does  previous  use  of  a  scenario  affect  your  selection? 


Value 


15.  Please  list  below  any  other  factors  that  effect  your  scenario  selection. 


FACTOR: 


Value 


FACTOR: 


Value 


FACTOR: 


Value 


FACTOR: 


Value 


Additional  Comments: 


Date 


Name  (optional)  _ _  Branch _  Rank 

Current  Duty  Position _ 

Previous  duty  positions  over  the  last  five  years _ 


All  of  these  questions  refer  to  factors  that  may  or  may  not  influence  your  selection  of  training 
scenarios  to  be  utilized  in  virtual  environment  such  as  SIMNET  or  CCTT.  Please  answer  them  by 
placing  a  numerical  value  corresponding  to  your  level  of  belief  that  you  would  use  each  factor  in 
selecting  a  training  scenario.  You  may  use  any  value  between  0  and  100.  The  scale  below  is  meant 
only  as  a  guide. 


Very  Small  Chance  Small  Chance  Could  Effect  High  Chance  Very  High  Chance 

0-20%  21-40%  41-60%  61-80%  81-100% 

1.  How  much  does  the  type  of  terrain  (dessert,  rolling  hills,  restrictive  etc.)  affect  your  selection? 

Value 


2.  How  much  does  the  light  condition  (daylight,  night,  %  illumination,  etc.)  affect  your  selection? 


Value 

3.  How  much  does  one  particular  task  affect  your  selection? 

Value 


4.  How  much  does  the  simulator  selection  (CCTT,  SIMNET  etc.)  affect  your  selection? 

Value  [ 


5.  How  much  does  the  friendly  unit’s  composition  (pure  or  combined  arms)  affect  your  selection? 

Value 


6.  How  much  does  the  enemy  unit’s  composition/capabilities  (motorized,  light,  armored,  attack 
aviation  etc.)  affect  your  selection? 

Value 


7.  How  much  does  the  enemy/friendly  ratio  affect  your  selection? 


Value 
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8.  How  much  does  the  existence  of  pre-planned  fires  affect  your  selection? 


Value 


9.  How  much  does  enemy  equipment  (T-62  vs.  T-72  or  Hind  D  vs.  Holkum)  affect  your  selection? 


Value 


10.  How  much  does  friendly  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache  Longbow)  affect  your 
selection? 

Value 


11.  How  much  does  friendly  bordering  unit  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache 
Longbow)  affect  your  selection? 

Value  I 


12.  How  much  does  enemy’s  training  level  affect  your  selection? 

Value 


13.  How  much  does  enemy’s  strength  level  affect  your  selection? 

Value 


14.  How  much  does  the  enemy’s  mission  affect  your  selection? 


Value 


15.  How  much  do  thermal  conditions  affect  your  selection? 


Value 


16.  How  much  does  fog,  rain  or  haze  affect  your  selection? 


Value 


17.  How  much  does  previous  use  of  a  scenario  affect  your  selection? 

Value 
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18.  Please  list  below  any  other  factors  that  effect  your  scenario  selection. 


FACTOR: 


FACTOR: 


FACTOR: 


FACTOR: 


Value 


Value 


Value 


Value 


ADDITIONAL  COMMENTS: 


Appendix  E 
Survey’s  Final  Form 
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The  following  questionnaire  is  designed  to  determine  what  factors  affect  a 
commander’s  or  operations  officer’s  selection  of  a  simulation  training  scenario.  The 
Standard  Army  Training  System-TRaining  Exercise  Development  System  (SATS- 
TREDS)  currently  selects  scenarios  based  on  the  desired  MTP  tasks  that  the  commander 
wants  trained.  These  selected  tasks  determine  the  training  audience.  The  intent  is  to 
determine  which  additional  factors  or  properties  are  the  most  important  for  virtual 
simulations,  such  as  SIMNET,  and  which  are  most  important  for  constructive 
simulations,  such  as  JANUS  or  CBS.  A  general  definition  of  a  virtual  training 
simulation  is  a  training  device  where  the  trainee  is  immersed  in  a  computer  generated 
environment  designed  to  replicate  conditions  and  react  to  stimuli  accurately  mimicking 
the  real  world  (battlespace).  A  constructive  training  simulation  can  be  defined  as 
computer  generated  people,  units,  and  equipment  operating  in  a  synthetic  battle,  where 
the  trainee(s)  interacts  through  computer  terminals,  screens  and  other  similar  devices. 
The  trainees  may  be  insulated  from  the  computer  architecture  by  means  of  organic  unit 
reporting  systems. 

The  most  important  factors  will  be  modeled  in  an  expert  system  within  SATS- 
TREDS.  As  part  of  SATS-TREDS,  the  commander  will  be  able  to  tailor  the  factors  to 
aid  in  selecting  the  appropriate  training  scenario  from  the  Army’s  simulation  library. 

The  selection  process  will  be  interactive;  the  commander  will  be  presented  with  5  or  6 
scenarios  that  closely  meet  his  needs.  Each  choice’s  strengths  will  be  highlighted  to  help 
the  commander  make  his  decision. 

Indicate  the  impact  each  factor  has  on  the  environment  by  placing  values  for 
virtual  simulations  in  the  triangle  and  the  responses  for  constructive  simulations  in  the 
square. 


Value 


Value 


Virtual  Constructive 

Please  respond  for  the  two  types  of  simulations:  virtual  (SIMNET,  COFT,  etc.) 
and  constructive  (Janus,  BCTP,  etc.).  Your  responses  are  independent  for  each 
category.  For  example:  there  is  no  reason  you  cannot  assign  a  score  of  “100”  to 
both  types  of  simulations  for  the  factor  “Terrain”. 
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Please  answer  the  following  questions.  All  information  will  be  kept  in  strict  confidence.  Only  aggregate 
data  will  be  reported. 


A.  General 

Name  (optional): 

Branch:  Rank: 

Unit  Current  Dutv  Position: 

Last  Level  of  Command: 

Military  Education  Level:  Age: 

Time  in  Service: 

Time  in  Unit _  Previous  duty  positions  over  the  last  five  years: 


B.  Training  and  Simulation 

For  the  following  questions,  circle  the  most  appropriate  response. 

1 .  If  you  related  your  unit  training  experience  in  terms  of  years,  where  would  you  place  yourself  on  the 

chart  below  ?  _ _ _ _ _ 

less  than  5  6-10  11-15  16-21  over  21 

years  years  years  years  years 

2.  In  the  last  year,  how  much  total  time  have  you/your  staff  spent  planning  simulation  based  training? 


less  than  10  11-20  21  -30  31  -40  over  41 

hours  hours  hours  hours  hours 

3.  In  the  last  year,  how  much  time  has  your  unit  spent  executing  simulation  based  training? 


less  than 
1  day 


1-5 

days 


6-10 

days 


10-15 

days 


over  16 
days 
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4.  Of  your  simulator  training  time,  how  much  is  in  a  virtual  environment  (SIMNET,  UCOFT,  Flight 
Trainer,  etc.)? 


less  than  1-5  6-10  10-15  over  16 

1  day  days  days  days  days 

5.  Of  your  simulator  training  time,  how  much  is  in  a  constructive  environment  (JANUS,  BBS,  CSSTSS, 
CBS,  etc.)? 


less  than 
1  day 


1-5 

days 


6-10 

days 


10-15 

days 


over  16 
days 


6.  How  effective  do  you  believe  simulation  based  training  is  in  preparing  your  unit  for  war? 


Not  effective  Somewhat  ineffective  Effective  Highly  effective  Very  highly  effective 

0-20%  21-40%  41-60%  61-80%  81-100% 

7.  Which  simulations  or  simulator  devices  have  you  or  your  unit  trained  with,  and  how  would  you 
evaluate  their  effectiveness?  (Please  check) 


Simulator 

I  or  my  unit  have 
trained  with  this 
simulator 

Not  effective 
0-20% 

Somewhat 

ineffective 

21-40% 

Effective 

41-60% 

Highly 

effective 

61-80% 

Very  Highly 
Effective 
81-100% 

agts  ; 

BBS 

CBS 

CCTT 

Driver  Trainer 

JANUS 

JCM 

MTS 

Pilot  Trainer 

SIMNET 

GUARDFIST  I 

TSFO 

UCCATS 

UCOFT 

OTHERS: 
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ADDITIONAL  COMMENTS: 


All  of  these  questions  refer  to  factors  that  may  or  may  not  influence  your  selection  of  training 
scenarios  to  be  utilized  in  constructive  and  virtual  simulations.  Please  answer  them  by  placing  a 
numerical  value  corresponding  to  your  level  of  belief  that  you  would  use  each  factor  in  selecting  a 
training  scenario.  You  may  use  any  value  between  0  and  100.  The  scale  below  is  meant  only  as  a 
guide.  You  are  to  give  answers  for  both  virtual  and  constructive  environments.  It  may  help  to 
imagine  that  you  are  planning  virtual  (SIMNET,  UCOFT,  PGT,  CCTT,  Pilot  trainer,  etc.)  and 
constructive  (JANUS,  BBS,  CBS,  etc.)  training  exercises.  What  are  the  chances  you  consider  these 
factors? 


Very  Small  Chance  Small  Chance  Could  Effect  High  Chance  Very  High  Chance 

0-20%  21-40%  41-60%  61-80%  81-100% 

1 .  How  much  does  the  type  of  terrain  (desert,  rolling  hills,  restrictive,  etc.)  affect  your  scenario 
selection? 


Value  /\ 

Virtual 


Value 

Constructive 


2.  How  much  does  one  particular  task  to  be  trained  (individual  through  collective)  affect  your 
scenario  selection? 


Value 


A 

Virtual 


Value 

Constructive 


3.  How  much  does  the  simulator  type  (JANUS,  CBS,  BBS,  SIMNET,  CCTT  etc.)  affect  your 
scenario  selection? 


Value 


Value 


Virtual  Constructive 

4.  How  much  does  the  friendly  unit’s  composition  (pure  or  combined  arms)  affect  your  scenario 
selection? 


Value 


A 

Virtual 


Value 

Constructive 
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9.  How  much  does  friendly  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache  Longbow)  affect  your 
scenario  selection? 

Value  /\  Value  |  | 

Virtual  Constructive 
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10.  How  much  does  adjacent  friendly  unit  equipment  (M60  vs.  M1A2  or  Cobra  vs.  Apache 
Longbow)  affect  your  scenario  selection? 


Value 


A 

Virtual 


Value 


Constructive 


Very  Small  Chance 
0-20% 


Small  Chance 
21-40% 


Could  Effect 
41-60% 


High  Chance 
61-80% 


1 1 .  How  much  does  the  enemy’s  training  level  affect  your  scenario  selection? 

A 


Value 


Value 


Virtual 


Constructive 


12.  How  much  does  the  enemy’s  mission  affect  your  scenario  selection? 


Value 


Value 


Virtual 


Constructive 


13.  How  much  does  weather  (fog,  rain,  or  haze)  affect  your  scenario  selection? 

Value 


Value 

Virtual 


Constructive 


14.  How  much  does  previous  use  of  a  scenario  affect  your  scenario  selection? 


Value 


Value 


Virtual 


Constructive 


15.  How  does  the  level  of  difficulty  of  the  exercise  affect  your  scenario  selection? 


Value 


Value 


Virtual 


Constructive 


Very  High  Chance 
81-100% 
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16.  How  much  does  unit  past  performance  assessment  (T’s,  P’s,  and  U’s)  affect  you  scenario 
selection? 


Value 


Value 


Virtual  Constructive 


Very  Small  Chance  Small  Chance  Could  Effect  High  Chance  Very  High  Chance 

0-20%  21-40%  41-60%  61-80%  81-100% 

17.  How  much  do  light  conditions  (daylight,  night,  %  illumination,  etc.)  affect  your  scenario 
selection? 

Value  Value  |  | 

Virtual  Constructive 


18.  How  much  does  the  capability  to  support  observation  devices  (FLIR,  Thermal,  NVG,  Optic,  etc.) 
and  their  degradation  affect  your  scenario  selection? 


Value 


Virtual 


Value 

Constructive 


19.  How  much  does  the  amount  of  exercise  preparation  required  (placement  of  forces  on  the  battle 
field,  coordination  of  simulation  facility  use,  and  other  administrative  tasks  associated  with  the 
training)  affect  your  scenario  selection? 


Value 

Virtual 


Value 

Constructive 


21.  How  much  does  the  amount  of  mission  planning  required  (troop  leading  procedures,  course  of 
action  development,  OPORD  preparation)  affect  your  scenario  selection? 


Value  /\ 
Virtual 


Value 

Constructive 
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Very  Small  Chance 
0-20% 


Small  Chance 
21-40% 


Could  Effect 
41-60% 


High  Chance 
61-80% 


Very  High  Chance 
81-100% 


Please  list  below  any  other  factors  that  affect  your 

FACTOR: 

scenario  selection? 

Value 

A 

Value 

— | 

Virtual 

Constructive 

FACTOR: 

Value 

A 

Value 

— | 

Virtual 

Constructive 

FACTOR: _ 

_ 

Value 

A 

Value 

| 

Virtual 

Constructive 

Additional  Comments: 


Appendix  F 
Tabulated  Pilot  Data 
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Questionnaire: 


Virtual  Responses 


1: 

70  i 

60 

65 

55; 

40 

65! 

80; 

65 

80; 

60: 

55 

75 

70; 

80; 

65 

65! 

65; 

65 

75! 

80 

2! 

75  i 

57 

90 

40! 

20 

25! 

20; 

90 

95; 

95  j 

85 

65 

100; 

90! 

100 

75; 

65! 

95 

10; 

5 

3: 

100  i 

100 

i  bo 

100! 

100 

100; 

100! 

100 

100; 

50! 

100 

100 

100! 

100! 

100 

100; 

0; 

50 

100; 

100 

4 : 

55; 

0 

70 

80 ; 

90 

90; 

50! 

80 

100; 

80! 

85 

100 

100! 

70; 

100 

100! 

99; 

100 

100! 

100 

5! 

80! 

70 

. 90 

10; 

90 

40! 

0; 

80 

20: 

15; 

15 

30 

70 

70! 

25 

30; 

75! 

30 

20; 

70 

6; 

100! 

80 

90 

90! 

100 

100; 

80! 

20 

60; 

40! 

100 

20 

100 

80; 

100 

90; 

100; 

90 

80; 

20 

7: 

80  i 

41 

100 

50; 

60 

81 

40! 

40 

90; 

70! 

80 

80 

90 

50; 

90 

50! 

70; 

50 

60! 

100 

8! 

30! 

60 

90 

10; 

40 

25 

20; 

16 

25i 

50j 

40 

45 

25 

25! 

40 

50; 

65! 

40 

70; 

50 

9: 

80! 

100 
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Appendix  G 

Pilot  Mean  Data  and  ANOVA 
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One-Way  Analysis  of  Variance 
All  Pilot  Responses 
Analysis  of  Variance 
Source  DF  SS  MS  F  p 
Factor  39  81212  2082  2.97  0.000 

Error  1160  812717  701 

Total  1199  893929 

Individual  95%  CIs  For  Mean 

Based  on  Pooled  StDev 


Level 

N 

Mean 

Terrain 

30 

75.73 

Task 

30 

53.17 

Simtor 

30 

66.83 

FComp 

30 

58.90 

EComp 

30 

64.43 

PowRat 

30 

65.43 

ETskOrg 

30 

51.53 

EEquip 

30 

62.03 

FEquip 

30 

69.40 

AdUnit 

30 

57.47 

ETmLev 

30 

62.10 

EMision 

30 

56.93 

Weather 

30 

73.03 

PrevUse 

30 

70.33 

Diff 

30 

68.87 

Assment 

30 

57.97 

LtData 

30 

62.40 

ObDev 

30 

66.77 

ExPrep 

30 

51.80 

MnPlan 

30 

62.70 

CTem 

30 

47.00 

CTask 

30 

54.33 

CSimlatr 

30 

42.47 

CFComp 

30 

43.67 

CEComp 

30 

61.33 

CPowRat 

30 

63.57 

CETaskOg 

30 

61.43 

CEEquip 

30 

62.77 

CFEquip 

30 

47.60 

CAdUnit 

30 

46.03 

CETranLv 

30 

56.30 

CEMssion 

30 

56.43 

C  Weather 

30 

48.73 

CPrevUse 

30 

54.17 

CDifclty 

30 

53.03 

CAssmnt 

30 

51.23 

CLtData 

30 

54.40 

CObsDev 

30 

44.80 

CExPrep 

30 

63.90 

CMsPlan 

30 

52.77 

Pooled  StDev  = 

26. 

47 

StDev 

— + - + - 

17.18 

( — 

25.08 

( — * — .) 

26.56 

( — * — > 

28.59 

( — * — ) 

28.54 

( — * — > 

26.33 

( — * — > 

27.20 

( — * — ) 

26.92 

c — * — ) 

21.91 

( — * — .) 

21.96 

( — * — ) 

23.83 

( — * — ) 

26.78 

( — *---) 

26.96 

( — * — 

23.78 

( — * — ) 

22.64 

( — 

25.43 

( — * — ) 

28.10 

( — * — ) 

20.96 

( — * — ) 

29.62 

( — * — ) 

27.91 

( — * — ) 

25.28 

(- 

— ) 

23.44 

( — * — ) 

29.02 

(-- 

--* — -) 

27.76 

(-- 

-* — ) 

26.71 

( — * — ) 

26.64 

( — * — .) 

22.87 

( — * — ) 

27.46 

( — * — ) 

29.53 

(- 

— * — ) 

26.55 

(- 

— ) 

24.41 

( — 

26.94 

( — * — ) 

30.43 

( 

; - * - .) 

28.93 

( - *---) 

28.43 

( - * - .) 

28.81 

( - 

29.29 

( - * - ) 

29.16 

(- 

— * - ) 

24.92 

( - * - ) 

29.86 

( - * - .) 

.+ - + - +. 

45  60  75 
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Single  Environment  Calculations  (z  test) 
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z-Test:  "1  wo  Sample  tor  Means 

wtmnmasuTmazminw 

observation  Devices 

Mean 

63.5666666/ 

-  TO? 

Known  Variance 

709.42 

850.3 

Observations 

30 

30 

Hypothesized  Mean  Difference 

c 

z 

2.60270200/1 

P(Z<=z)  one-tail 

0.00462465 

z  Critical  one-tail 

P(Z<=z)  two-tail 

z  critical  two-tan 

1.959961082 

z  >  z  critical 
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Observation  Devices 

80 

30 

50 

60 

0 

50 

Chi-Squared 

Test 

Critical  value  for  6- 1-2=3  df  and  alpha  = 
Conclusion:  Accept  Ho  (data  is  normal) 

Bin  Frequenc 

y 

5%  is  7.815 

Expected 

Frequency 

Chi-Square 

terms 

50 

0.00  25.00  8 

5.5902 

0.4523 

35 

25.00  40.00  4 

5.5818 

0.3603 

80 

40.00  53.00  7 

5.2829 

0.0152 

70 

53.00  69.00  5 

5.5794 

0.0317 

60 

69.00  100.00  6 

5.2235 

5.2235 

0 

100.00  More  0 

6.0830 

50 

30 

40 

Observation  Devices 

61 

50 

Mean 

44.8 

68 

Standard  Error 

5.323856523 

85 

Median 

50 

15 

Mode 

50 

50 

Standard  Deviation 

29.1599631 

90 

Sample  Variance 

850.3034483 

10 

Kurtosis 

-0.882601457 

100 

Skewness 

-0.101795785 

0 

Range 

100 

5 

Minimum 

0 

60 

Maximum 

100 

0 

Sum 

1344 

50 

Count 

30 

15 

Confidence  Level(95.0%) 

10.88851511 
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Power  Ratio  Chi-Squared  Test 


95 

30 

65 

50 

55 

80 

Critical  value  for  6- 1-2=3  df  and  alpha  —  5%  is 

7.815 

Conclusion:  Accept  Ho  (data  is  normal) 

Expected  Chi-Square 
Bin  Frequency  Frequency  terms 

80 

0.00  40.00 

5  5.3889 

0.0281 

80 

40.00  55.00 

5  5.5709 

0.0585 

60 

55.00  67.00 

4  5.3215 

0.3282 

0 

67.00  81.00 

11  5.7693 

4.7424 

80 

81.00  100.00 

5  5.1219 

0.0029 

50 

100.00  More 

0 

5.1600 

10 

13 

90 

Power  Ratio 

80 

70 

Mean 

63.56667 

74 

Standard  Error 

4.862874 

80 

Median 

72 

60 

Mode 

#NUM! 

20 

Standard  Deviation 

26.63506 

100 

Sample  Variance 

709.4264 

50 

Kurtosis 

0.212669 

90 

Skewness 

-0.917163 

100 

Range 

100 

75 

Minimum 

0 

75 

Maximum 

100 

55 

Sum 

1907 

80 

Count 

30 

60 

Confidence 

9.9457 

Level(95.0%) 

159 


Appendix  I 


ANOVA  for  Armor  (Pilot) 
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One-Way  Analysis  of  Variance 

Armor 

Analysis  of  Variance 

Source  DF  SS  MS  F  p 

Factor  39  22761  584  1.04  0.426 

Error  120  67482  562 

Total  159  90243 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level  N 

Mean 

StDev 

- +___. 

— + - + - 

Terrain 

4 

67.50 

25.00 

( — * — ) 

Task 

4 

55.25 

34.01 

( — * — ) 

Simtor 

4 

80.00 

21.60 

( — * — ) 

FComp 

4 

37.50 

22.17  (■ 

— ■ 

— * - ) 

EComp 

4 

50.00 

11.55 

c 

— * — > 

PowRat 

4 

46.50 

29.08 

(- 

— * — ) 

ETskOrg 

4 

30.00 

18.26  (-- 

.... 

-* — ) 

EEquip 

4 

36.50 

18.50  (- 

... 

...» — ) 

FEquip 

4 

61.25 

27.20 

( — * — ) 

AdUnit 

4 

55.00 

12.91 

( — * — ) 

ETmLev 

4 

61.25 

17.50 

( — ».._..„) 

EMision 

4 

58.75 

25.29 

( — » — > 

Weather 

4 

73.75 

32.50 

( — » — ) 

PrevUse 

4 

46.25 

14.93 

(- 

— * — ) 

Diff  4 

62.50 

22.17 

( — 

. — > 

Assment 

4 

50.00 

16.33 

c 

— » — > 

LtData 

4 

68.75 

16.52 

( — » — ) 

ObDev 

4 

47.50 

5.00 

(-■ 

— ». — ) 

ExPrep 

4 

48.75 

21.75 

(- 

— » — ) 

MnPlan 

4 

71.25 

27.80 

( — * — ) 

CTem 

4 

60.00 

32.66 

( — » — ) 

CTask 

4 

63.75 

14.93 

( — » — ) 

CSimlatr 

4 

57.75 

18.17 

( — * — ) 

CFComp 

4 

65.25 

26.65 

( — » — > 

CEComp 

4 

59.00 

30.99 

( — * — ) 

CPowRat 

4 

80.00 

16.33 

( — * — > 

CETaskOg 

4 

71.50 

17.67 

( — * — ) 

CEEquip 

4 

65.00 

38.73 

( — » — ) 

CFEquip 

4 

56.25 

21.36 

( — » — ) 

CAdUnit 

4 

55.00 

21.21 

( — » — ) 

CETranLv 

4 

71.25 

23.23 

( — » — ) 

CEMssion 

4 

51.25 

25.29 

( 

— » — ) 

C  Weather 

4 

56.25 

43.08 

( — » — ) 

CPrevUse 

4 

82.75 

2.63 

( — * — > 

CDifclty 

4 

80.25 

16.44 

( — » — > 

CAssmnt 

4 

61.25 

30.65 

( — * — ) 

CLtData 

4 

56.25 

26.89 

( — » — ) 

CObsDev 

4 

63.75 

25.62 

( — * — ) 

CExPrep 

4 

70.00 

9.13 

( — » — ) 

CMsPlan 

4 

52.50 

25.98 

( — » — > 

- + - +.... 

Pooled  StDev 

=  23.71 

30  60 
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ANOVA  for  Infantry  (Pilot) 
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One-Way  Analysis  of  Variance 
Infantry 

Analysis  of  Variance 
Source  DF  SS  MS  F  p 
Factor  39  40833  1047  1.36  0.104 

Error  120  92174  768 

Total  159  133007 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 

Level  N  Mean  StDev  - + - +- 


Terrain 

4 

81.25 

Task 

4 

59.25 

Simtor 

4 

78.75 

FComp 

4 

55.00 

EComp 

4 

46.75 

PowRat 

4 

72.50 

ETskOrg 

4 

70.00 

EEquip 

4 

88.75 

FEquip 

4 

88.75 

AdUnit 

4 

72.50 

ETmLev 

4 

72.50 

EMision 

4 

70.25 

Weather 

4 

88.75 

PrevUse 

4 

88.75 

Diff 

4 

83.75 

Assment 

4 

66.25 

LtData 

4 

37.50 

ObDev 

4 

62.00 

ExPrep 

4 

65.00 

MnPlan 

4 

65.00 

CTem 

4 

40.00 

CTask 

4 

63.75 

CSimlatr 

4 

33.50 

CFComp 

4 

37.75 

CEComp 

4 

67.50 

CPowRat 

4 

52.50 

CETaskOg 

4 

77.50 

CEEquip 

4 

65.00 

CFEquip 

4 

37.50 

CAdUnit 

4 

47.50 

CETranLv 

4 

52.50 

CEMssion 

4 

57.50 

CWeather 

4 

50.00 

CPrevUse 

4 

55.00 

CDifclty 

4 

45.00 

CAssmnt 

4 

50.00 

CLtData 

4 

55.00 

CObsDev 

4 

52.50 

CExPrep 

4 

47.50 

CMsPlan 

4 

35.00 

Pooled  StDev  = 

27.71 

13.15  ( - * - ) 

32.69  ( - * - ) 

19.31  ( - * - ) 

32.40  ( - * - ) 

36.45  ( - * - ) 

35.71  ( - * - ) 

34.64  ( - * - ) 

16.52  ( - * - ) 

10.31  ( - * - ) 

21.02  ( - * - ) 

23.98  ( - * - ) 

24.43  ( - * - ) 

14.36  ( - * - ) 

8.54  ( - * - ) 

18.87  ( - * - ) 

31.19  ( - * - ) 

32.79  ( - * - ) 

24.62  ( - * - ) 

38.51  ( - * - ) 

41.43  ( - * - ) 

23.09  ( - * - ) 

25.62  ( - * - ) 

38.54  ( - * - ) 

10.34  ( - * - ) 

32.02  ( - * - ) 

34.28  ( - * - ) 

12.58  ( - * - ) 

12.91  ( - * - ) 

34.03  ( - * - ) 

15.00  ( - * - ) 

35.94  ( - * - ) 

37.75  ( - * - ) 

29.44  ( - * - ) 

31.09  ( - * - ) 

28.87  ( - * - ) 

14.14  ( - * - ) 

26.46  ( - * - ) 

20.62  ( - * - ) 

29.86  ( - * - ) 

38.51  ( - * - ) 

- + - + - +- 

30  60  90 
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Chi-square  test  of  Frequency  Independence  For  Terrain  and  Task  (Pilot) 


Chi-Square  Test 
Terrain 


Expected  counts  are  printed  below  observed  counts 


V 

.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi ' 

1 

1 

2 

1 

9 

17  30 

5.00 

3.00 

6.00 

6.00 

10.00 

2 

9 

4 

11 

3 

3  30 

5.00 

3.00 

6.00 

6.00 

10.00 

Total 

10 

6 

12 

12 

20  60 

ChiSq  =  3.200+  0.333  +  4.167+  1.500+  4.900  + 
3.200+  0.333  +  4.167+  1.500+  4.900  =  28.200 
df=4,  p  =  0.000 

2  cells  with  expected  counts  less  than  5.0 


Chi-Square  Test 
Task 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

3 

3 

12 

9  3 

30 

2.50 

6.00 

10.50 

8.50 

2.50 

2 

2 

9 

9 

8  2 

30 

2.50 

6.00 

10.50 

8.50 

2.50 

Total 

5 

12 

21 

17 

5  60 

ChiSq  =  0.100+  1.500+  0.214+  0.029+  0.100  + 
0.100+  1.500+  0.214+  0.029+  0.100  =  3.887 
df  =  4,  p  =  0.422 

4  cells  with  expected  counts  less  than  5.0 
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Tabulated  Data  for  the  Target  Population 
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Constructive  Responses 
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Mean/ANOVA  Data  for  Target  Population  by  Sub-category 


One-Way  Analysis  of  Variance 
All  Respondents 
Analysis  of  Variance 

Source  DF  SS  MS  F  p 
Factor  39  101672  2607  3.25  0.000 

Error  2194  1757575  801 

Total  2233  1859247 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 

N 

Mean 

StDev  - 

— - 

■J- 

■ 

■ 

i 

i 

i 

i 

i 

i 

f 

i 

i 

i 

i 

■ 

i 

i 

i 

f 

C  Terrai 

62 

64.48 

26.49 

( — *_) 

C  Task 

62 

68.34 

22.14 

( — 

C  Simtor 

60 

67.00 

25.92 

( — * — ) 

C  FComp 

62 

65.48 

29.16 

C  EComp 

62 

65.90 

29.28 

C  PowRat 

62 

64.69 

25.47 

C  ETkOrg 

62 

57.98 

31.80 

C  EEquip 

62 

58.47 

29.39 

C  FEquip 

62 

66.02 

25.66 

CAdUnit 

62 

57.65 

26.93 

C  ETmLv 

62 

51.82 

30.60 

(-- 

-* — ) 

C  EMison 

62 

65.69 

27.90 

( — * — ) 

C  Wether 

62 

51.47 

30.76 

(-- 

C  PreUse 

62 

70.42 

22.34 

CDiff 

61 

66.30 

26.76 

( — * — > 

C  Assmnt 

61 

77.75 

21.08 

( - * - ' 

C  LtData 

62 

49.60 

31.78  ( 

— - 

C  ObDev 

62 

48.31 

31.12  (- 

-* — ) 

C  ExPrep 

62 

66.77 

28.42 

C  MnPlan 

62 

62.58 

31.73 

V  Terran 

50 

65.54 

27.42 

( — *__) 

V  Task 

50 

66.38 

25.10 

( — * — ) 

V  Sim 

49 

62.86 

28.70 

( — * — ) 

V  FComp 

50 

53.60 

30.70 

(- 

- * - ) 

V  EComp 

50 

61.56 

28.69 

— ) 

V  Ratio 

50 

58.36 

24.75 

( — * — ) 

VETaskO 

50 

51.42 

31.88 

(- 

— * — ) 

V  EEquip 

50 

55.78 

29.95 

< 

: — * — ) 

V  FEquip 

50 

64.36 

25.32 

( — * — ) 

VAdjUnt 

50 

55.12 

27.19 

(■ 

— * — ) 

V  Tmlvl 

50 

51.72 

31.92 

(-- 

-* — ) 

VEMsn 

50 

60.16 

30.24 

( — * — ) 

V  Wether 

50 

58.70 

29.68 

( — * — ) 

VPrevU 

50 

65.70 

25.42 

( — 

VLvlDif 

50 

63.14 

26.67 

( — * — ) 

V  UtAsmt 

49 

75.71 

27.28 

( - * - 

VLtDta 

50 

56.96 

29.61 

(---* - ) 

V  ObDev 

50 

61.40 

30.94 

(---* - ) 

VExPrp 

50 

58.80 

30.58 

( - * - ) 

V  MsnPln 

50 

57.80 

31.29 

(--* - ) 

- + - + - 

Pooled  StDev  = 

28.30 

48 

60  72  84 

84 


Appendix  N 


Mean/ANOVA  Data  for  the  Air  Defense  Population 
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One-Way  Analysis  of  Variance 
Air  Defense 
Analysis  of  Variance 
Source  DF  SS  MS  F  p 
Factor  39  17320  444  0.55  0.982 

Error  120  96432  804 

Total  159  113752 

Individual  95%  CIs  For  Mean 


Level 

N 

Based  on 
Mean 

C  Terrai 

4 

81.25 

C  Task 

4 

57.50 

C  Simtor 

4 

62.50 

C  FComp 

4 

67.50 

C  EComp 

4 

65.00 

C  PowRat 

4 

67.50 

C  ETkOrg 

4 

61.25 

C  EEquip 

4 

61.25 

C  FEquip 

4 

73.75 

CAdUnit 

4 

55.00 

C  ETmLv 

4 

60.00 

C  EMison 

4 

62.50 

C  Wether 

4 

50.00 

C  PreUse 

4 

87.50 

CDiff 

4 

82.50 

C  Assmnt 

4 

90.75 

C  LtData 

4 

46.25 

C  ObDev 

4 

52.50 

C  ExPrep 

4 

57.50 

C  MnPlan 

4 

61.25 

V  Terr  an 

4 

68.75 

V  Task 

4 

55.00 

V  Sim 

4 

48.75 

V  FComp 

4 

63.75 

V  EComp 

4 

63.75 

V  Ratio 

4 

62.50 

VETaskO 

4 

53.75 

V  EEquip 

4 

56.25 

V  FEquip 

4 

71.25 

VAdjUnt 

4 

50.00 

V  Tmlvl 

4 

57.50 

VEMsn 

4 

57.50 

V  Wether 

4 

52.50 

V  PrevU 

4 

72.50 

VLvlDif 

4 

68.75 

V  UtAsmt 

4 

75.75 

VLtDta 

4 

53.75 

V  ObDev 

4 

61.25 

VExPrp 

4 

50.00 

VMsnPln 

4 

61.25 

Pooled  StDev  = 

28.35 

Pooled  StDev 

StDev  — h - + - +— 

15.48  ( - * - ) 

26.30  ( - * - ) 

27.54  ( - * - ) 

15.00  ( - * - ) 

17.32  ( - * - ) 

20.62  ( - * - ) 

29.55  ( - * - ) 

29.55  ( - * - ) 

17.02  ( - * - ) 

28.87  ( - 111 - ) 

31.62  ( - * - ) 

34.03  ( - * - ) 

35.59  ( - * - ) 

8.66  ( - * - ) 

2.89  ( - * - ) 

9.78  ( - * - ) 

28.69  ( - * - ) 

37.75  ( - * - ) 

35.94  ( - * - ) 

39.66  ( - * - ) 

33.26  ( - * - ) 

20.82  ( - * - ) 

33.26  ( - * - ) 

29.26  ( - * - ) 

17.97  ( - * - ) 

20.62  ( - * - ) 

30.38  ( - * - ) 

28.10  ( - * - ) 

21.75  ( - * - ) 

25.82  ( - * - ) 

33.04  ( - * - ) 

33.04  ( - * - ) 

27.54  ( - * - ) 

22.55  ( - * - ) 

32.76  ( - * - ) 

37.76  ( - * - ) 

28.10  ( - * - ) 

37.05  ( - * - ) 

33.67  ( - * - ) 

39.66  ( - * - ) 

— +- - + - + — 
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Appendix  O 


Mean/ANOVA  Data  for  the  Armor  Population 
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One-Way  Analysis  of  Variance 
Armor 

Analysis  of  Variance 

Source  DF  SS  MS 

F 

P 

Factor  39 

48958 

1255 

1.84 

0.003 

Error  260 
Total  299 

Level  N 

176989 

225947 

Mean 

681 

Individual  95%  CIs  For  Mean 

Based  on  Pooled  StDev 

StDev  - + - + - + - 

C  Terrai 

8 

50.62 

23.97 

C  Task 

8 

65.00 

27.26 

C  Simtor 

8 

71.25 

32.71 

( — *_) 

C  FComp 

8 

57.62 

29.65 

( — * — ) 

C  EComp 

8 

47.62 

29.22 

(--*—) 

C  PowRat 

8 

57.00 

24.36 

( — * — ) 

C  ETkOrg 

8 

48.87 

22.55 

C  EEquip 

8 

44.38 

27.96 

( — * — ) 

C  FEquip 

8 

53.75 

30.68 

CAdUnit 

8 

50.00 

27.26 

( — * — ) 

CETmLv 

8 

36.37 

19.84 

C  EMison 

8 

49.50 

25.37 

( — * — ) 

C  Wether 

8 

33.25 

30.57  ( - * - ) 

C  PreUse 

8 

71.87 

22.35 

CDiff 

8 

62.50 

33.27 

( — 

C  Assmnt 

8 

78.13 

17.31 

( — *__) 

C  LtData 

8 

37.50 

27.65 

( — * — ) 

C  ObDev 

8 

45.63 

24.41 

( — * — ) 

C  ExPrep 

8 

70.62 

24.56 

( — 

C  MnPlan 

8 

53.75 

34.20 

( — * — > 

V  Terran 

7 

77.14 

9.51 

( — 

V  Task 

7 

71.57 

26.78 

( — 

V  Sim 

7 

80.00 

11.55 

( — * — ; 

V  FComp 

7 

54.43 

33.62 

(---* — ) 

V  EComp 

7 

53.00 

31.51 

(- — * — ) 

V  Ratio 

7 

55.86 

26.40 

( — * — ) 

VETaskO 

7 

58.00 

28.57 

( — * — ) 

V  EEquip 

7 

47.29 

30.47 

( — *--) 

V  FEquip 

7 

56.57 

30.40 

( — * — ) 

VAdjUnt 

7 

52.29 

26.80 

( — * — ) 

VTmlvl 

7 

42.86 

22.89 

( — * — ) 

VEMsn 

7 

50.14 

25.63 

( — * — ) 

V  Wether 

7 

58.00 

26.58 

(-— * — ) 

V  PrevU 

7 

75.00 

22.17 

VLvlDif 

7 

75.00 

15.55 

( — * — ) 

VUtAsmt 

7 

89.29 

6.07 

( — 

VLtDta 

7 

65.00 

21.41 

( — * — ) 

V  ObDev 

7 

68.57 

17.73 

( — *--) 

VExPrp 

7 

61.43 

31.32 

( — * — ) 

VMsnPln 

7 

62.86 

28.12 

( — * — ) 

Pooled  StDev 

-  26.09 

- +- - +- - 

30  60  90 

Mean/ANOVA  Data  for  the  Aviation  Population 
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One-Way  Analysis  of  Variance 
Aviation 

Analysis  of  Variance 


Source 

DF 

SS 

MS 

F 

P 

Factor 

39 

28721 

736 

0.94 

0.577 

Error 

200 

156746 

784 

Total 

239 

185467 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 

N 

Mean 

StDev 

— +_ - + - +- 

C  Terrai 

6 

51.67 

38.04 

( — * — ) 

C  Task 

6 

59.17 

20.10 

(- — * — ) 

C  Simtor 

6 

65.00 

26.65 

(- — * — ) 

C  FComp 

6 

58.33 

30.61 

( — * — ) 

C  EComp 

6 

74.17 

12.81 

( — * — ) 

C  PowRat 

6 

78.33 

14.72 

c — *- — ) 

C  ETkOrg 

6 

58.33 

31.89 

( — * — ) 

C  EEquip 

6 

50.00 

29.66 

( — * — ) 

C  FEquip 

6 

61.67 

21.37 

(- — * — ) 

C  AdUnit 

6 

60.00 

24.49 

( — * — } 

CETmLv 

6 

53.33 

25.82 

( — * — ) 

C  EMison 

6 

71.67 

27.87 

( — * — ) 

C  Wether 

6 

63.33 

28.05 

( — * — ) 

C  PreUse 

6 

80.00 

20.00 

( — * — > 

CDiff 

6 

58.33 

27.87 

( — * — ) 

C  Assmnt 

6 

68.33 

21.37 

( — * — ) 

C  LtData 

6 

43.33 

32.66  ( 

— * — ) 

C  ObDev 

6 

40.83 

36.66  (- 

— * — > 

C  ExPrep 

6 

85.00 

13.78 

( — * — ) 

C  MnPlan 

6 

73.33 

32.04 

( — * — ) 

V  Terran 

6 

60.00 

35.21 

( — * — ) 

V  Task 

6 

67.50 

23.61 

( — * — ) 

V  Sim 

6 

68.33 

28.58 

( — * — ) 

V  FComp 

6 

54.17 

36.93 

( — * — ) 

V  EComp 

6 

74.17  ‘ 

13.57 

( — * — ) 

V  Ratio 

6 

66.67 

24.22 

( — * — ) 

VETaskO 

6 

51.67 

33.12 

( — * — ) 

V  EEquip 

6 

61.67 

35.45 

( — * — ) 

V  FEquip 

6 

80.00 

26.83 

( — * — ) 

VAdjUnt 

6 

71.67 

30.61 

( — * — ) 

VTmlvl 

6 

65.00 

31.46 

( — * — ) 

VEMsn 

6 

65.00 

33.32 

( — * — ) 

V  Wether 

6 

86.67 

15.06 

( — * — ) 

V  PrevU 

6 

76.67 

29.44 

( — * — ) 

VLvlDif 

6 

58.33 

27.87 

( — * — } 

V  UtAsmt 

6 

60.00 

32.25 

(- — * — ) 

V  LtDta 

6 

68.33 

29.27 

( — *- — ) 

V  ObDev 

6 

89.17 

12.01 

( — * — ) 

VExPrp 

6 

63.33 

32.66 

c — * — ) 

VMsnPln 

6 

65.00 

34.50 

(- — * — ) 

- -h- - +- - H - +- 

Pooled  StDev  =  28.00  30  60  90  120 
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Mean/ANOVA  Data  for  the  Field  Artillery  Population 
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One-Way  Analysis  of  Variance 
Field  Artillery 
Analysis  of  Variance 

Source  DF  SS  MS  F  p 
Factor  39  28225  724  0.94  0.577 

Error  238  183279  770 

Total  277  211504 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 

N 

Mean 

StDev 

i 

i 

i 

i 

i 

■ 

i 

i 

f 

i 

i 

i 

i 

i 

i 

i 

i 

t 

t 

C  Terrai 

7 

67.14 

33.40 

( — * — ) 

C  Task 

7 

86.43 

18.42 

( — *. — ) 

C  Simtor 

6 

60.83 

34.12 

( — * — > 

C  FComp 

7 

85.00 

22.55 

( — * — ) 

C  EComp 

7 

88.57 

14.64 

( — * — > 

C  PowRat 

7 

73.00 

19.82 

( — * — ) 

C  ETkOrg 

7 

68.71 

33.93 

( — * — ) 

C  EEquip 

7 

81.57 

14.63 

( — * — > 

C  FEquip 

7 

74.43 

24.44 

( — * — ) 

CAdUnit 

7 

64.43 

26.47 

( — * — ) 

C  ETmLv 

7 

74.29 

24.23 

( — * — ) 

C  EMison 

7 

80.71 

26.37 

( — * — > 

C  Wether 

7 

61.43 

40.59 

( — * — ) 

C  PreUse 

7 

70.00 

28.43 

( — * — ) 

CDiff 

7 

88.57 

9.00 

( — * — > 

C  Assmnt 

7 

94.00 

7.66 

( — * — ) 

C  LtData 

7 

53.57 

40.07 

( — * — > 

C  ObDev 

7 

66.43 

34.97 

( — * — ) 

C  ExPrep 

7 

77.43 

27.68 

( — * — ) 

CMnPlan 

7 

87.86 

14.68 

( — * — ) 

V  Terran 

7 

75.00 

33.79 

( — * — ) 

V  Task 

7 

85.71 

15.12 

( — * — > 

V  Sim 

6 

50.83 

44.09  ( 

— * — ) 

V  FComp 

7 

64.43 

36.52 

( — * — ) 

V  EComp 

7 

70.00 

33.67 

( — * — ) 

V  Ratio 

7 

67.29 

19.87 

( — * — ) 

V  ETaskO 

7 

64.43 

34.17 

( — * — ) 

V  EEquip 

7 

77.29 

12.58 

( — * — ) 

V  FEquip 

7 

70.86 

21.76 

( — * — ) 

VAdjUnt 

7 

62.29 

24.59 

( — * — ) 

VTmlvl 

7 

72.14 

25.80 

( — * — ) 

VEMsn 

7 

66.43 

38.48 

( — * — ) 

V  Wether 

7 

70.00 

39.16 

( — * — ) 

V  PrevU 

7 

70.00 

28.43 

( — * — ) 

VLvlDif 

7 

87.14 

7.56 

( — * — > 

V  UtAsmt 

7 

89.71 

15.07 

( — * — ) 

VLtDta 

7 

70.71 

35.41 

( — * — ) 

V  ObDev 

7 

84.29 

18.35 

( — * — ) 

VExPrp 

7 

68.86 

35.06 

( — * — > 

VMsnPln 

7 

76.43 

28.39 

( — * — ) 

i 

i 

i 

i 

i 

i 

i 

i 

f 

i 

i 

t 

t 

i 

i 

i 

i 

+ 

Pooled  StDev  = 

27.75 

50  75  100 
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Mean/ANOVA  Data  for  the  Infantry  Population 
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One-Way  Analysis  of  Variance 
Infantry 

Analysis  of  Variance 

Source  DF  SS  MS  F  p 
Factor  39  50388  1292  1.55  0.020 

Error  478  397869  832 

Total  517  448257 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 

N 

Mean 

StDev 

- + - + - + — 

C  Terrai 

16 

72.50 

20.17 

( — * — ) 

C  Task 

16 

73.75 

15.55 

( — * — ) 

C  Simtor 

15 

64.33 

26.31 

( — * — ) 

C  FComp 

16 

66.56 

33.05 

( — * — ) 

C  EComp 

16 

61.56 

35.44 

( — * — ) 

C  PowRat 

16 

67.81 

25.03 

( — * — ) 

C  ETkOrg 

16 

55.63 

36.28 

( — * — ) 

C  EEquip 

16 

59.56 

31.94 

( — * — } 

C  FEquip 

16 

69.25 

24.10 

( — * — ) 

C  AdUnit 

16 

56.25 

31.22 

( — * — ) 

CETmLv 

16 

45.69 

33.06 

( 

— * — > 

C  EMison 

16 

61.56 

30.97 

( — * — ) 

C  Wether 

16 

44.06 

25.11 

(■ 

— * — ) 

C  PreUse 

16 

60.94 

20.18 

( — * — > 

CDiff 

15 

61.73 

25.14 

( — * — > 

C  Assmnt 

16 

69.69 

24.18 

( — * — ) 

C  LtData 

16 

50.00 

32.30 

( — * — ) 

C  ObDev 

16 

41.56 

29.20 

(- 

— ) 

C  ExPrep 

16 

53.12 

29.49 

( — * — ) 

C  MnPlan 

16 

57.44 

32.47 

( — * — ) 

V  Terran 

10 

71.50 

20.15 

( — * — ) 

V  Task 

10 

66.10 

24.26 

( — * — ) 

V  Sim 

10 

67.50 

28.10 

( — * — j 

V  FComp 

10 

48.60 

34.88 

(- 

— * — ) 

V  EComp 

10 

54.00 

32.98 

( — * — ) 

V  Ratio 

10 

59.00 

21.32 

( — * — ) 

V  ETaskO 

10 

42.50 

34.58 

(~ 

— * — ) 

V  EEquip 

10 

44.50 

31.31 

(-- 

— * — ) 

V  FEquip 

10 

59.00 

21.32 

( — * — ) 

VAdjUnt 

10 

47.00 

31.29 

(- 

— * — ) 

V  Tmlvl 

10 

40.50 

34.36 

(-- 

---* — ) 

VEMsn 

10 

54.00 

34.38 

( — * — ) 

V  Wether 

10 

41.00 

26.01 

(~ 

— * — ) 

V  PrevU 

10 

58.00 

22.01 

( — * — ) 

V  LvlDif 

10 

58.60 

30.96 

( — * — ) 

V  UtAsmt 

10 

61.00 

38.06 

( — * — ) 

VLtDta 

10 

53.00 

26.69 

( — * — ) 

V  ObDev 

10 

52.60 

30.18 

( — * — ) 

VExPrp 

10 

35.80 

21.85 

( — 

-* — ) 

VMsnPln 

10 

42.90 

30.76 

(~ 

— * — ) 

— ■ 

4 

+ 

i 

i 

i 

i 

■ 

■ 

i 

i 

i 

■ 

i 

i 

i 

i 

i 

i 

i 

+ 

Pooled  StDev  = 

28.85 

20  40  60  80 
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Mean/ANOVA  Data  for  the  Military  Police  Population 


One-Way  Analysis  of  Variance 
Military  Police 
Analysis  of  Variance 


Source 

DF 

SS 

MS 

F 

P 

Factor 

39 

31181 

800 

1.12 

0.306 

Error 

140 

99595 

711 

Total 

179 

130777 

Individual  95%  CIs  For  Mean 


Based  on  Pooled  StDev 

Level  N  Mean  StDev  - + - + - +- 

CTerrai  5  49.20  31.00  ( - * - ) 

C  Task  5  70.00  18.71  ( - * - ) 

CSimtor  5  71.00  22.47  ( - * - ) 

C  FComp  5  60.00  33.91  ( - * - ) 

C  EComp  5  49.20  32.72  ( - * - ) 

CPowRat  5  50.20  32.06  ( - * - ) 

C  ETkOrg  5  49.20  37.02  ( - * - ) 

CEEquip  5  46.20  25.44  ( - * - ) 

CFEquip  5  50.20  21.57  ( - * - ) 

CAdUnit  5  52.00  21.68  ( - * - ) 

CETmLv  5  43.00  32.33  ( - * - ) 

C  EMison  5  54.00  23.82  ( - * - ) 

C  Wether  5  44.00  29.87  ( - * - ) 

C  PreUse  5  51.00  23.56  ( - * - ) 

C  Diff  5  48.00  22.80  ( - * - ) 

C  Assmnt  5  87.60  8.29  ( - * - ) 

CLtData  5  42.00  34.02  ( - * - ) 

C  ObDev  5  38.00  30.54  ( - * - ) 

CExPrep  5  72.00  19.87  ( - * - ) 

CMnPlan  5  58.00  33.65  ( - * - ) 

V  Terran  4  28.75  24.62  ( - * - ) 

VTask  4  45.25  24.02  ( - * - ) 

V  Sim  4  40.00  14.14  ( - * - ) 

V  FComp  4  37.50  18.93  ( - * - ) 

V  EComp  4  50.25  24.90  ( - * - ) 

V  Ratio  4  46.50  30.75  ( - * - ) 

VETaskO  4  35.25  33.62  ( - * - ) 

VEEquip  4  50.25  29.78  ( - * - ) 

VFEquip  4  50.25  24.90  ( - * - ) 

VAdjUnt  4  52.50  25.00  ( - * - ) 

VTmlvl  4  35.00  31.09  ( - * - ) 

VEMsn  4  50.00  25.50  ( - * - ) 

V  Wether  4  36.25  28.10  ( - * - ) 

VPrevU  4  51.25  27.20  ( - * - ) 

VLvlDif  4  45.00  23.80  ( - * - ) 

VUtAsmt  4  84.50  5.26  (<- - * - ) 

VLtDta  4  31.25  27.80  ( - * - ) 

V  ObDev  4  28.75  25.94  ( - * - ) 

VExPrp  4  68.75  17.02  ( - * - ) 

VMsnPln  4  47.50  27.84  ( - * - ) 

- + - + - +- 

Pooled  StDev  =  26.67  30  60  90 


198 


Appendix  T 


Mean/ANOVA  Data  for  the  Quarter  Master  Population 
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One-Way  Analysis  of  Variance 


Quartermaster 
Analysis  of  Variance 


Source 

DF 

SS 

MS 

F 

P 

Factor 

39 

22148 

568 

0.88 

0.658 

Error 

60 

38646 

644 

Total 

99 

60794 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 
C  Terrai 
C  Task 
C  Simtor 
C  FComp 
C  EComp 
C  PowRat 
C  ETkOrg 
C  EEquip 
C  FEquip 
C  AdUnit 
CETmLv 
C  EMison 
C  Wether 
C  PreUse 
CDiff 
C  Assmnt 
C  LtData 
C  ObDev 
C  ExPrep 
C  MnPlan 

V  Terr  an 

V  Task 

V  Sim 

V  FComp 

V  EComp 

V  Ratio 

V  ETaskO 

V  EEquip 

V  FEquip 
VAdjUnt 

V  Tmlvl 
VEMsn 

V  Wether 
VPrevU 
VLvlDif 

V  UtAsmt 
VLtDta 

V  ObDev 
VExPrp 
VMsnPln 

Pooled  StDev  = 


N  Mean  StDev  --H - + - +■ 


3 

76.67 

2.89 

( — * — ) 

3 

56.67 

25.17 

( — * — ) 

3 

69.67 

27.02 

( — * — ) 

3 

81.33 

18.04 

( — * — ) 

3 

89.67 

0.58 

( — * — ) 

3 

52.00 

32.51 

( — * — ) 

3 

61.33 

35.92 

( — * — ) 

3 

64.00 

38.57 

( — * — ) 

3 

83.33 

5.77 

( — * — ) 

3 

81.00 

1.73 

( — * — ) 

3 

65.00 

39.69 

( — * — ) 

3 

77.00 

24.02 

( — * — ) 

3 

79.67 

26.08 

( — * — ) 

3 

85.33 

8.39 

( — * — ) 

3 

68.33 

38.84 

( — * — ) 

3 

76.00 

16.37 

( — * — ) 

3 

76.00 

6.93 

( — * — ) 

3 

61.33 

16.29 

( — * — ) 

3 

89.00 

8.54 

( — * — ) 

3 

59.67 

44.16 

( — » — ) 

2 

72.50 

3.54 

( — * — ) 

2 

40.00 

28.28 

( 

— * — ) 

2 

68.00 

31.11 

( — * — ) 

2 

50.50 

0.71 

( — * — ) 

2 

85.00 

7.07 

( — * — ) 

2 

35.00 

21.21 

(- 

— * — ) 

2 

50.50 

43.13 

( — * — ) 

2 

54.00 

48.08 

( — * — ) 

2 

82.50 

10.61 

( — * — ) 

2 

79.00 

1.41 

( — * — ) 

2 

55.00 

49.50 

( — * — ) 

2 

70.50 

28.99 

( — * — ) 

2 

74.00 

33.94 

( — * — ) 

2 

65.00 

21.21 

( — * — ) 

2 

36.50 

19.09 

(- 

— * — ) 

2 

70.00 

28.28 

( — * — ) 

2 

75.50 

6.36 

( — * — ) 

2 

51.00 

1.41 

( — * — ) 

2 

92.00 

2.83 

( — * — ) 

2 

30.00 

28.28 

(-■ 

— * — ) 

+  +  + 

25.38 

_ 

0 

40  80  120 

.+■ 
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Mean/ANOVA  Data  for  the  Ground  Maneuver  Population 


One-Way  Analysis  of  Variance 
Ground  Maneuver 
Analysis  of  Variance 

Source  DF  SS  MS  F  p 
Factor  39  70752  1814  2.34  0.000 

Error  778  603723  776 

Total  817  674475 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level 

N 

Mean 

StDev 

— +~ 

i 

i 

i 

1 

I 

l 

+ 

i 

■ 

I 

I 

i 

I 

1 

i 

i 

+ 

I 

I 

I 

■ 

i 

i 

I 

1 

+ 

i 

C  Terrai 

24 

65.21 

23.47 

( - * - ) 

C  Task 

24 

70.83 

20.04 

( - * - ) 

C  Simtor 

23 

66.74 

28.15 

( - * - ) 

C  FComp 

24 

63.58 

31.60 

( - * - ) 

C  EComp 

24 

56.92 

33.52 

( - * - ) 

C  PowRat 

24 

64.21 

24.83 

( - * - ) 

CETkOrg 

24 

53.37 

32.00 

( — * — ) 

C  EEquip 

24 

54.50 

30.93 

( - * - ) 

C  FEquip 

24 

64.08 

26.85 

( - * - ) 

C  AdUnit 

24 

54.17 

29.51 

( - * - ) 

CETmLv 

24 

42.58 

29.20 

(-- 

-* - ) 

C  EMison 

24 

57.54 

29.24 

( - * - ) 

C  Wether 

24 

40.46 

26.89 

( — 

.* - ) 

C  PreUse 

24 

64.58 

21.11 

( - * - ) 

CDiff 

23 

62.00 

27.47 

( - * - ) 

C  Assmnt 

24 

72.50 

22.12 

( - * - ) 

C  LtData 

24 

45.83 

30.81 

(~ 

- * - ) 

C  ObDev 

24 

42.92 

27.22 

(-- 

- ) 

C  ExPrep 

24 

58.96 

28.67 

( - * - ) 

CMnPlan 

24 

56.21 

32.35 

( - * - ) 

VTerran 

17 

73.82 

16.44 

( - * - ) 

V  Task 

17 

68.35 

24.65 

( - * - ) 

V  Sim 

17 

72.65 

23.11 

( - * - ) 

V  FComp 

17 

51.00 

33.42 

(- 

- * - ) 

V  EComp 

17 

53.59 

31.38 

( 

- * - ) 

V  Ratio 

17 

57.71 

22.79 

( - * - ) 

VETaskO 

17 

48.88 

32.26 

(- 

- *- - ) 

V  EEquip 

17 

45.65 

30.02 

(-- 

— * - ) 

V  FEquip 

17 

58.00 

24.57 

( - * - ) 

VAdjUnt 

17 

49.18 

28.76 

(~ 

- * - ) 

VTmlvl 

17 

41.47 

29.36 

( — 

-* - ) 

VEMsn 

17 

52.41 

30.25 

(- 

- * - ) 

V  Wether 

17 

48.00 

26.83 

(-- 

-  * - ) 

V  PrevU 

17 

65.00 

23.05 

( - * - ) 

VLvlDif 

17 

65.35 

26.44 

( - * - ) 

V  UtAsmt 

17 

72.65 

32.17 

( - * - ) 

VLtDta 

17 

57.94 

24.69 

( - * - ) 

V  ObDev 

17 

59.18 

26.38 

( - * - ) 

V  ExPip 

17 

46.35 

28.38 

(-- 

—* - ) 

VMsnPln 

17 

51.12 

30.52 

(- 

- * - ) 

— +. 

Pooled  StDev  = 

27.86 

32 

48  64  80 
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Mean/ANOVA  Data  for  the  Combat  Support  and  Combat  Service  Support  Population 


One-Way  Analysis  of  Variance 

Combat  Support  and  Combat  Service  Support 

Analysis  of  Variance 

Source  DF  SS  MS  F  p 
Factor  39  59737  1532  2.01  0.000 

Error  700  534718  764 

Total  739  594455 

Individual  95%  CIs  For  Mean 
Based  on  Pooled  StDev 


Level  N 

Mean 

StDev  - 

— + - 

- + - +. - +- 

C  Terrai 

21 

65.57 

24.43 

C  Task 

21 

61.67 

21.81 

C  Simtor 

21 

70.90 

22.02 

(--* — ) 

C  FComp 

21 

61.38 

28.02 

( — * — ) 

C  EComp 

21 

64.29 

28.41 

( — * — ) 

C  PowRat 

21 

56.57 

28.69 

(---* — ) 

C  ETkOrg 

21 

56.71 

31.97 

( — * — ) 

C  EEquip 

21 

54.71 

27.85 

( — * — ) 

C  FEquip 

21 

65.29 

25.20 

CAdUnit 

21 

54.67 

24.86 

CETmLv 

21 

51.43 

32.29 

( — * — ) 

C  EMison 

21 

66.48 

26.80 

( — * — ) 

C  Wether 

21 

54.48 

31.75 

( — * — ) 

C  PreUse 

21 

74.57 

22.14 

( — * — ) 

CDiff 

21 

66.67 

26.89 

( — * — ) 

C  Assmnt 

21 

82.33 

16.78 

( — * — ) 

C  LtData 

21 

53.52 

30.42 

(---* — ) 

C  ObDev 

21 

49.24 

31.31 

( — * — ) 

C  ExPrep 

21 

68.19 

29.53 

C  MnPlan 

21 

59.52 

32.43 

( — * — ) 

V  Terran 

16 

55.63 

28.74 

( — * — } 

V  Task 

16 

54.44 

24.02 

( — * — ) 

V  Sim 

16 

55.38 

25.95 

( — * — } 

V  FComp 

16 

48.19 

24.63 

( — * — > 

V  EComp 

16 

59.75 

27.95 

( — * — ) 

V  Ratio 

16 

49.44 

27.45 

( — * — ) 

VETaskO 

16 

44.25 

30.88  ( - * - ) 

V  EEquip 

16 

51.56 

29.41 

( — * — ) 

V  FEquip 

16 

63.19 

25.92 

( — 

VAdjUnt 

16 

51.12 

25.18 

( — * — > 

V  Trill  vl 

16 

45.31 

33.49 

( — * — ) 

VEMsn 

16 

60.06 

28.00 

( — * — ) 

V  Wether 

16 

52.06 

26.60 

( — * — ) 

VPrevU 

16 

61.88 

27.50 

( — * — ) 

VLvlDif 

16 

52.37 

26.03 

( — * — ) 

V  UtAsmt 

16 

80.06 

19.52 

( — * — ) 

VLtDta 

16 

46.37 

28.58 

( — * — ) 

V  ObDev 

16 

44.87 

31.94 

( — * — ) 

VExPrp 

16 

65.56 

29.54 

( — * — > 

V  MsnPln 

16 

54.37 

31.83 

( — * — ) 

Pooled  StDev 

=  27.64 

40  60  80 
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Comparison  of  Factors  to  Constructive  Terrain  Mean  (z  test) 


Constructive  Terrain  vs.  Constructive  Enemy  Task  Organization 
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Constructive  Terrain  vs.  Constructive  Enemy  Training  Level 


207 


C  Weathe 


Constructive  Terrain  vs.  Constructive  Weather 
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C  Terrain 


Constructive  Terrain  vs.  Constructive  Light  Data 


209 


Observation  Device 


Constructive  Terrain  vs.  Constructive  Observation  Devices 
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C  Terrain 


Constructive  Terrain  vs.  Virtual  Friendly  Composition 
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V  Enemy  Task  O 


Constructive  Terrain  vs.  Virtual  Enemy  Task  Organization 
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Constructive  Terrain  vs.  Virtual  Enemy  Equipment 
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Adacent  Uni 


Constructive  Terrain  vs.  Virtual  Adjacent  Unit 
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V  E  Train  Lev 


Constructive  Terrain  vs.  Virtual  Enemy  Training  Level 
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Appendix  X 


Constructive  vs.  Virtual  Factor  Analysis  (z  test) 


216 


Terrain 


VTas 


Task 


218 


C  Simulator 


Simulator 


219 


C  F  Com 


Friendly  Unit  Composition 


220 


C  E  Com 


Enemy  Composition 


221 


Combat  Power  Ratio 


C  Enemy  Task  Org 


223 


CEE 


Enemy  Equipment 


224 


Friendly  Equipment 


225 


Adjacent  Unit 


Train  Lev 


Enemy  Training  Level 


227 


V  E  Missi 


228 


Weather 


V  Previous  Use 


230 


V  Difficult 


Level  of  Difficulty 


231 


Unit  Assessment 


232 


Light  Dat 


Light  Data 


233 


Observation  Devices 


V  Exercise  Pre 


235 


Mission  Planni 


Mission  Planning 


236 


Appendix  Y 


Constructive  vs.  Virtual  Factor  Analysis  (chi-square) 


237 


Chi-Square  Test 
Adjacent  Unit 

Expected  counts  are  printed  below  observed  counts 


V. 

Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  CH  Total 

1 

8 

8 

22 

11 

13  62 

8.30 

9.96 

19.38 

13.29 

11.07 

2 

7 

10 

13 

13 

7  50 

6.70 

8.04 

15.62 

10.71 

8.93 

Total 

15 

18 

35 

24 

20  112 

ChiSq 

=  0.011  +  0. 

387  + 

0.356  + 

0.393  +  0.336  + 

0.014+  0.480+  0.441  +  0.488+  0.417  =  3.322 
df  =  4,  p  =  0.506 


Chi-Square  Test 
Unit  Assesment 

Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 
1  1  4  8  19  29  61 


3.33 

3.88 

6.65 

15.53 

31.61 

2 

5 

3 

4 

9  28 

49 

2.67 

3.12 

5.35 

12.47 

25.39 

Total 

6 

7 

12 

28 

57  1 

ChiSq  =  1.628+  0.004+  0.272+  0.777+  0.215  + 
2.026+  0.004+  0.339+  0.967+  0.268  =  6.500 
df  =  4,  p  =  0.166 

4  cells  with  expected  counts  less  than  5.0 
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Chi-Square  Test 
Level  of  Difficulty 


Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 
1  5  5  17  13  21  61 


5.50 

6.59 

16.49 

12.64 

19.78 

2 

5 

7 

13 

10 

15  50 

4.50 

5.41 

13.51 

10.36 

16.22 

Total 

10 

12 

30 

23 

36  111 

ChiSq 

=  0.045  +  0. 

386  + 

0.016  + 

0.010+  0.075  + 

0.055  +  0.470+  0.020+  0.013  +  0.091  =  1.179 
df  =  4,  p  =  0.881 

1  cells  with  expected  counts  less  than  5.0 


Chi-Square  Test 
Enemy  Composition 


Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  CH 

1  6 

10 

9 

13 

24  62 

6.09 

11.07 

9.41 

14.39 

21.04 

2  5 

10 

8 

13 

14  50 

4.91 

8.93 

7.59 

11.61 

16.96 

Total  1 1 

20 

17 

26 

38  112 

ChiSq  =  0.001  +  0.104+  0.018+  0.135  +  0.418  + 
0.002+  0.129+  0.022+  0.167+  0.518  =  1.513 
df  =  4,  p  =  0.824 

1  cells  with  expected  counts  less  than  5.0 
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Chi-Square  Test 
Enemy  Equipment 

Expected  counts  are  printed  below  observed  counts 


V 

.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi 

1 

9 

8 

17 

11 

17  62 

10.52 

6.64 

16.05 

13.84 

14.95 

2 

10 

4 

12 

14 

10  50 

8.48 

5.36 

12.95 

11.16 

12.05 

Total 

19 

12 

29 

25 

27  112 

ChiSq  =  0.219+  0.277+  0.056+  0.583  +  0.282  + 
0.272+  0.344+  0.069+  0.722+  0.350  =  3.174 
df  =  4,  p  =  0.530 

Chi-Square  Test 
Enemy  Mission 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

6 

7 

11 

18 

20 

62 

7.75 

7.75 

10.52 

17.71 

18.27 

2 

8 

7 

8 

14 

13 

50 

6.25 

6.25 

8.48 

14.29 

14.73 

Total 

14 

14 

19 

32 

33 

112 

ChiSq  =  0.395+  0.073  +  0.022+  0.005  +  0.164  + 
0.490+  0.090+  0.027+  0.006+  0.204=  1.475 
df  =  4,  p  =  0.831 
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Chi-Square  Test 
Enemy  Task  Organization 

Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 

1  12  6  11  14  19  62 

13.29  6.64  12.73  12.73  16.61 

2  12  6  12  9  11  50 

10.71  5.36  10.27  10.27  13.39 


Total  24  12  23  23  30  112 


ChiSq  =  0.124+  0.062+  0.236+  0.126+  0.345  + 
0.154+  0.077+  0.292+  0.157+  0.428  =  2.001 
df  =  4,  p  =  0.736 


Chi-Square  Test 
Enemy  Training  Level 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

13 

10 

15 

14 

10  62 

14.39 

9.41 

13.29 

13.84 

11.07 

2 

13 

7 

9 

11 

10  50 

11.61 

7.59 

10.71 

11.16 

8.93 

Total 

26 

17 

24 

25 

20  112 

ChiSq  =  0.135  +  0.037+  0.221  +  0.002+  0.104  + 
0.167+  0.046+  0.274+  0.002+  0.129=  1.117 
df  =  4,  p  =  0.892 
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Chi-Square  Test 
Exercise  Preparation 


Expected  counts  are  printed  below  observed  counts 


Y.  Sm  CH  Sm.  CH  C.  Effct 

Hi  CH  V.  Hi  CH 

1  6 

8 

7 

21 

20  62 

8.30 

9.96 

7.20 

17.71 

18.82 

2  9 

10 

6 

11 

14  50 

6.70 

8.04 

5.80 

14.29 

15.18 

Total  15 

18 

13 

32 

34  112 

ChiSq  =  0.639+  0.387+  0.005+  0.609+  0.074  + 
0.792+  0.480+  0.007+  0.756+  0.092  =  3.841 
df  =  4,  p  =  0.428 

Chi-Square  Test 
Friendly  Composition 


Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm.  CH  C.  Effct 

Hi  CH  Y.  Hi  CH 

1  8 

6 

12 

17 

19  62 

10.52 

8.30 

11.63 

14.39  17.16 

2  11 

9 

9 

9 

12  50 

8.48 

6.70 

9.37 

11.61 

13.84 

Total  19 

15 

21 

26 

31  112 

ChiSq  =  0.603  +  0.639+  0.012+  0.472+  0.197  + 
0.747+  0.792+  0.015  +  0.586+  0.244  =  4.308 
df  =  4,  p  =  0.367 
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Chi-Square  Test 
Friendly  Equipment 


Expected  counts  are  printed  below  observed  counts 


V 

.  Sm  CH  Sm 

.  CH  C. 

Effct 

Hi  CH  V.  Hi  CH 

1 

4 

7 

17 

15 

19  62 

4.43 

7.20 

16.05 

16.61 

17.71 

2 

4 

6 

12 

15 

13  50 

3.57 

5.80 

12.95 

13.39 

14.29 

Total 

8 

13 

29 

30 

32  112 

ChiSq  =  0.041  +  0.005+  0.056+  0.156+  0.093  + 
0.051  +  0.007+  0.069+  0.193  +  0.116  =  0.787 
df  =  4,  p  =  0.940 

2  cells  with  expected  counts  less  than  5.0 


Chi-Square  Test 
Light  Data 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 

1  18  5  16  13  10  62 

15.50  4.98  16.61  12.73  12.18 

2  10  4  14  10  12  50 

12.50  4.02  13.39  10.27  9.82 

Total  28  9  30  23  22  112 

ChiSq  =  0.403  +  0.000  +  0.022  +  0.006  +  0.390  + 
0.500+  0.000+  0.028+  0.007+  0.483  =  1.839 
df  =  4,  p  =  0.765 

2  cells  with  expected  counts  less  than  5.0 


Chi-Square  Test 
Mission  Planning 
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Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  CH 

1  9 

9 

9 

14 

21  62 

10.52 

8.86 

10.52 

12.73  19.38 

2  10 

7 

10 

9 

14  50 

8.48 

7.14 

8.48 

10.27 

15.62 

Total  19 

16 

19 

23 

35  112 

ChiSq  =  0.219+  0.002+  0.219+  0.126+  0.136  + 
0.272+  0.003  +  0.272+  0.157+  0.169  =  1.575 
df  =  4,  p  =  0.813 


Chi-Square  Test 
Observation  Devices 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  Y.  Hi  CH  Total 


1 

16 

12 

14 

10 

10  62 

13.29 

9.96 

13.84 

10.52 

14.39 

2 

8 

6 

11 

9  16  50 

10.71 

8.04 

11.16 

8.48 

11.61 

Total 

24 

18 

25 

19 

26  112 

ChiSq  =  0.555  +  0.416+  0.002+  0.025  +  1.341  + 
0.688+  0.516+  0.002+  0.032+  1.663  =  5.238 
df  =  4,  p  =  0.265 
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Chi-Square  Test 
Combat  Power  Ratio 


Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

6 

5 

15 

17 

19  62 

6.09 

6.64 

18.27 

15.50 

15.50 

2 

5 

7 

18 

11 

9  50 

4.91 

5.36 

14.73 

12.50 

12.50 

Total 

11 

12 

33 

28 

28  112 

ChiSq 

=  0.001  +  0. 

406  + 

0.585  + 

0.145  +  0.790 

0.002+  0.504+  0.725+  0.180+  0.980  =  4.318 
df  =  4,p  =  0.365 

1  cells  with  expected  counts  less  than  5.0 

Chi-Square  Test 
Previous  Use 


Expected  counts  are  printed  below  observed  counts 


V 

.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  CH 

1 

2 

5 

16 

19 

20  62 

2.77 

7.75 

14.39 

18.27 

18.82 

2 

3 

9 

10 

14 

14  50 

2.23 

6.25 

11.61 

14.73 

15.18 

Total 

5 

14 

26 

33 

34  112 

ChiSq  =  0.213  +  0.976+  0.179+  0.029+  0.074  + 
0.264+  1.210+  0.223  +  0.036+  0.092  =  3.296 
df  =  4,  p  =  0.510 

2  cells  with  expected  counts  less  than  5.0 
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Chi-Square  Test 
Simulator 


Expected  counts  are  printed  below  observed  counts 


V 

.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  i 

1 

5 

9 

8 

21 

17  60 

6.61 

8.26 

9.36 

17.61 

18.17 

2 

7 

6 

9 

11 

16  49 

5.39 

6.74 

7.64 

14.39 

14.83 

Total 

12 

15 

17 

32 

33  109 

ChiSq  =  0.390+  0.067+  0.197+  0.651  +  0.075  + 
0.478  +  0.082  +  0.241  +  0.797  +  0.092  =  3.069 
df  =  4,  p  =  0.547 


Chi-Square  Test 
Task 


Expected  counts  are  printed  below  observed  counts 


V.  Sm  CH  Sm. 

CH  C. 

Effct 

Hi  CH  V.  Hi  CH 

1  3 

7 

11 

14 

15  50 

2.23 

5.80 

12.50 

16.07 

13.39 

2  2 

6 

17 

22 

15  62 

2.77 

7.20 

15.50 

19.93 

16.61 

Total  5 

13 

28 

36 

30  112 

ChiSq  =  0.264+  0.247+  0.180+  0.267+  0.193  + 
0.213  +  0.199+  0.145+  0.215  +  0.156  =  2.079 
df  =  4,  p  =  0.721 

2  cells  with  expected  counts  less  than  5.0 
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Chi-Square  Test 
Terrain 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

6 

7 

10 

24 

15  62 

6.64 

6.09 

9.96 

21.59 

17.71 

2 

6 

4 

8 

15 

17  50 

5.36 

4.91 

8.04 

17.41 

14.29 

Total 

12 

11 

18 

39 

32  112 

ChiSq  =  0.062+  0.136+  0.000+  0.269+  0.416  + 
0.077+  0.169+  0.000+  0.334+  0.516=  1.979 
df  =  4,  p  =  0.740 

1  cells  with  expected  counts  less  than  5.0 


Chi-Square  Test 
Weather 

Expected  counts  are  printed  below  observed  counts 

V.  Sm  CH  Sm.  CH  C.  Effct  Hi  CH  V.  Hi  CH  Total 


1 

13 

11 

16 

10 

12  62 

12.18 

7.75 

17.71 

10.52 

13.84 

2 

9 

3 

16 

9  13  50 

9.82 

6.25 

14.29 

8.48 

11.16 

Total 

22 

14 

32 

19 

25  112 

ChiSq  =  0.055  +  1.363  +  0.166+  0.025  +  0.244  + 
0.069+  1.690+  0.206+  0.032+  0.303  =4.153 
df  =  4,  p  =  0.386 
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Appendix  Z 


General  Maneuver  and  Combat  Service  and  Combat  Service  Support  Frequency  Tables 


i 
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Constructive  Factors 
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Appendix  AA 


Corrected  General  Maneuver  and  Combat  Service  and  Combat  Service  Support  Frequency 

Factor  Tables  and  Analysis 
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Revised  Frequency  Table 


Italic  Factors  are  Combat  Service  and  Combat  Service  Support 
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atistics  tor  constructive  i  errain 


mmmmm 


KK  (si 


Hum 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


arning: 


0.177 

0.179 

0.065 

0.034 

0.036 

0.036 


eiis  are  sparse 


atistics  tor  constructive  ras 


oerticie 


»  1 

>96 

^  i 

2  1 

127 

2  1 

atistics  tor  constructive  simulator 


ism  i 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


mmmm 


lium 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


arning:  More  tnan  1/5  ot 


f 


0.111 

0.111 

0.210 

0.105 

0.105 

0.106 


ea  ceils  are  sparse 


0.283 

0.144 

0.144 

0.144 
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atistics  tor  constructive  hrienaiy  composition 


c  •  value;  : 


earson  cm-bquare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Pm 

0.146 

Contigency 

0.145 

Cramer's  V 

0.146 

Goodman-Kruskal  Gamma 

-0.033 

0.258 

Kendalls  tau-b 

-0.018 

0.142 

Stuart's  tau-c 

-0.020 

0.154 

Somer's  D  (C|R) 

-0.020 

0.155 

aming:  Moretnan  vt>  ot  i-inea  ceils  are  sparse 


atistics  for  constructive  tnemy  composition 


earson  cm-bquare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


aming:  iviore 


0.213 

0.218 

0.219 

0.121 

0.130 

0.131 


ens  are  bparse 


atistics  tor  constructive  comoa 


earson 


i-bquare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


0.145 

0.146 

-0.226 

-0.126 

-0.138 

-0.139 


0.246 

0.141 

0.155 

0.156 


atistics  tor  constructive 


rgamzation 


earson  Chi-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


rUVl 


0.077 

0.404 


eiis  are  sparse 


atistics  tor  constructive  tnemy  tquipmen 


c  ii  value 


earson 


_  i-square 

Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


lisisn 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


0.244 

0.141 

0.161 

0.162 


atistics  tor  constructive  t-rienaiy  tquipmen 


ataeiii  ill 


earson  cm-bquare 
Continuity  Adjusted  Chi-Square  0.41 0 

Likelihood  Ratio  Chi-Square  1.171 


ymptotic 


Pm 

0.161 

Contigency 

0.159 

Cramer's  V 

0.161 

Goodman-Kruskal  Gamma 

0.268 

0.247 

Kendalls  tau-b 

0.148 

0.140 

Stuart's  tau-c 

0.162 

0.153 

Somer’s  D  (C|R) 

0.163 

0.153 
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Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


aming:  More  than  1/5  o 
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atistics  tor  constructive 


mmmmmrn 


earson  Chi-Square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


lisim 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


riKM 


0.202 

0.207 

0.082 

0.047 

0.053 

0.054 


0.245 

0.142 

0.160 

0.160 


atistics  tor  constructive  previous  use 


earson  cm-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


aming: 


.  1^0 

.256 

2 

.170 

2 

0. 
0.156 
0.158 
0.269 
0.140 
0.144 
0.145 


eiis  are  sparse 


symptotic 


0.265 

0.140 

0.146 

0.146 


atistics  tor  constructive  Level  o 


earson 


Mil 


i-bquare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


'  *  coefficient  ySm 

Contingency 

0.093 

Cramer's  V 

0.093 

Goodman-Kruskal  Gamma 

0.155 

0.259 

Kendalls  tau-b 

0.085 

0.143 

Stuart's  tau-c 

0.093 

0.157 

Somer's  D  (C|R) 

0.093 

0.157 
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atistics  tor  constructive  Assessmen 


C  ; | | ;;ij:  ...  VaJUO 


ymptptic 


0.297 

Contigency 

0.285 

Cramer’s  V 

0.297 

Goodman-Kruskal  Gamma 

0.604 

0.263 

Kendalls  tau-b 

0.268 

0.129 

Stuart's  tau-c 

0.229 

0.117 

Somer's  D  (C|R) 

0.230 

0.117 

-itted  Cells  are  Sparse 

atistics  tor  constructive  Lig 


mmrnmmm, 


earson  cm-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


lisim 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


.  10/1 

.503 

4 

2 

.138 

2 

0.157 

0.159 

0.257 

0.149 

0.172 

0.173 


ll’l&ll 


MM 

I 


igpnHrnBUCTiij 


servation  Devices 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


ItV&ttl 


0.815 

0.585 


pymptpuc 


0.T53 

Contigency 

0.151 

Cramer's  V 

0.153 

Goodman-Kruskal  Gamma 

0.214 

0.240 

Kendalls  tau-b 

0.122 

0.139 

Stuart's  tau-c 

0.136 

0.156 

Somer's  D  (C|R) 

0.137 

0.157 
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atistics  for  Constructive  txercise  Planning 


earson 


MST 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Wmrnm 


HUiTI 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


armng:  More 


0.198 

0.202 

0.149 

0.079 

0.083 

0.083 _ 


ens  are  sparse 


atistics  for  constructive  Mission  Planning 


atuegjiisiiii 


earson  Chi-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


lisim 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


0.210 

0.215 

0.000 

0.255 

0.000 

0.143 

0.000 

0.156 

0.000 

0.157 

0.341 

Contigency 

0.322 

Cramer's  V 

0.341 

Goodman-Kruskal  Gamma 

-0.545 

0.242 

Kendalls  tau-b 

-0.300 

0.154 

Stuart's  tau-c 

-0.309 

0.163 

Somer's  D  (C|R) 

-0.309 

0.163 
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Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


aming:  More 


0.263 

0.272 

-0.420 

-0.245 

-0.272 

-0.272 


ells  are  sparse 


imuiator 


Chi-Square 

i-Square 


0.253 

0.159 

0.177 

0.177 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


0.320 

0.337 

-0.543 

-0.319 

-0.349 

-0.349 


ens  are  sparse 


atistics  for  virtual  Frienaiy  composition 


c :  value :  ; 


earson 


i-square 

Continuity  Adjusted  Chi-Square  3.140 

Likelihood  Ratio  Chi-Square  5.578 


0.228 

0.154 

0.169 

0.169 


oetticien 


Contigency 

0.368 

Cramer's  V 

0.396 

Goodman-Kruskal  Gamma 

-0.097 

0.280 

Kendalls  tau-b 

-0.058 

0.169 

Stuart's  tau-c 

-0.066 

0.192 

Somer's  D  (C|R) 

-0.066 

0.192 
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atistics  tor  virtual  tmemy  composition 


aiue  ■  : 


earson  oni-bquare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Imm 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer’s  D  (C|R) 


armng:  More 


\VIA 


— 


0.123 

0.124 

0.114 

0.063 

0.070 

0.070 


eus  are  sparse 


0.297 

0.166 

0.183 

0.183 


ower  Katio 


earson  cm-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


IlSJiTI 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


2  1 

0.204 

2  1 

0.798 

2  1 

0.153 

0.155 

-0.180 

-0.105 

-0.121 

-0.121 


0.277 

0.164 

0.189 

0.189 


rgamzation 


earson 


i-t>quare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Phi 

0.160 

Contigency 

0.158 

Cramer's  V 

0.160 

Goodman-Kruskal  Gamma 

-0.184 

0.281 

Kendalls  tau-b 

-0.106 

0.164 

Stuart's  tau-c 

-0.121 

0.187 

Somer’s  D  (C|R) 

-0.121 

0.187 
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atistics  tor  virtual  bnemy  bquipmen 


i-t>quare 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


IllJiTl 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


nuin 


0.049 

0.050 

0.000 

0.000 

0.000 

0.000 


0.286 

0.165 

0.189 

0.189 


atistics  tor  virtual  i-riencny  bquipmen 


c  value 


earson  uni-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Phi 

0.152 

Contigency 

0.150 

Cramer’s  V 

0.152 

Goodman-Kruskal  Gamma 

0.225 

0.284 

Kendalls  tau-b 

0.128 

0.164 

Stuart's  tau-c 

0.143 

0.183 

Somer's  D  (C|R) 

0.143 

0.184 

eus  are  sparse 


IBMIB 


earson  um-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


1 


liliiilMa 


Phi 

0.087 

Contigency 

0.087 

Cramer's  V 

0.087 

Goodman-Kruskal  Gamma 

0.000 

0.285 

Kendalls  tau-b 

0.000 

0.164 

Stuart’s  tau-c 

0.000 

0.189 

Somer's  D  (C|R) 

0.000 

0.189 

Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Sculler's  D  (C|R) 


earson 


atistic 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


0.388 

0.421 

0.058 

0.034 

0.037 

0.037 


eus  are  sparse 


atistics  Tor  virtual  tnemy  Mission 


mmm i 


0.298 

0.172 

0.189 

0.189 


Phi 

(7.043 

Contigency 

0.043 

Cramer's  V 

0.043 

Goodman-Kruskal  Gamma 

0.065 

0.296 

Kendalls  tau-b 

0.036 

0.165 

Stuart’s  tau-c 

0.040 

0.184 

Someds  D  (C|R) 

0.040 

0.184 

Warning:  More  than  1/S  of  1 

-itted  Cells  are  Sparse 

i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


Phi 

Contigency 

0.217 

Cramer's  V 

0.222 

Goodman-Kruskal  Gamma 

0.071 

0.285 

Kendalls  tau-b 

0.042 

0.168 

Stuart's  tau-c 

0.048 

0.192 

Someds  D  (C|R) 

0.048 

0.192 
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ifnnra^rara*Piyint»Tnis 


revious  use 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


00 

2 

CO 

C\| 

mmmm 


hum 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


arning:  More  than  1 


mm 


mm Hm 


0.100 

0.101 

-0.122 

-0.067 

-0.073 

-0.074 


eus  are  sparse 


0.298 

0.166 

0.181 

0.181 


earson 


mu 


i-square 
Continuity  Adjusted  Chi-Square 
Likelihood  Ratio  Chi-Square 


ghsi 


aiue  = 


i. 
0.653 
1.643 


lum 


Contigency 
Cramer's  V 

Goodman-Kruskal  Gamma 
Kendalls  tau-b 
Stuart's  tau-c 
Somer's  D  (C|R) 


arning:  More  than 


[114*1 


0.217 

0.222 

-0.301 

-0.177 

-0.202 

-0.202 


eus  are  sparse 


0.265 

0.162 

0.184 

0.185 


sytjtptptlgj 


Phi 

Contigency 

0.235 

Cramer's  V 

0.242 

Goodman-Kruskal  Gamma 

0.495 

0.333 

Kendalls  tau-b 

0.214 

0.157 

Stuart's  tau-c 

0.180 

0.138 

Somer's  D  (C|R) 

0.180 

0.138 

263 


Table  Statistics  for  Virtual  Light  Data 

Test  Statistic 

Value  I  S  DFx  :  :  x 

proto,  'mm. 

Pearson  Chi-Square 

0.795  I 

0.672 

Continuity  Adjusted  Chi-Square 

0.204  2 

0.903 

Likelihood  Ratio  Chi-Square 

0.798  2 

0.671 

iicoetticient 

value  Asymptotic  std,  fcrror  | 

pm - “ - 

035 

Contigency 

0.153 

Cramer's  V 

0.155 

Goodman-Kruskal  Gamma 

-0.180 

0.277 

Kendalls  tau-b 

-0.105 

0.164 

Stuart's  tau-c 

-0.121 

0.189 

Somer's  D  (C|R) 

-0.121 

0.189 

Table  Statistics  for  Virtual  Observation  Devices 

; ‘ ;  i  i‘:  statistic 

Proto. 

Pearson  Chi-Square 

- TZZA - — 2 - 

0.283 

Continuity  Adjusted  Chi-Square 

1 .245  2 

0.537 

Likelihood  Ratio  Chi-Square 

2.560  2 

0.278 

Coefficient 

|! :  i  ii  yai  As^fnp tottc-'St U'f  *  terror- villi 

PHi 

077 - 

Contigency 

0.267 

Cramer's  V 

0.277 

Goodman-Kruskal  Gamma 

-0.404 

0.246 

Kendalls  tau-b 

-0.242 

0.158 

Stuart's  tau-c 

-0.279 

0.182 

Somer's  D  (C|R) 

-0.279 

0.182 

Warning:  More  than  1/5  of  Fitted  Cells  are  Sparse 


\~~~~  Table  Statistics  for  Virtual  Exercise  Preparation  \ 

Test  statistic 

:  Value  HDF 

Proto. 

Pearson  Chi-Square 

4.212  2 

'  . . 0.'122 

Continuity  Adjusted  Chi-Square 

2.821  2 

0.244 

Likelihood  Ratio  Chi-Square 

4.325  2 

0.115 

1|  Coefficient 

value  Asymptotic  sta.  Error  §|  | 

Phi  “ 

0357 

Contigency 

0.336 

Cramer's  V 

0.357 

Goodman-Kruskal  Gamma 

0.573 

0.217 

Kendalls  tau-b 

0.343 

0.155 

Stuart's  tau-c 

0.375 

0.169 

Somer's  D  (C|R) 

0.375 

0.169 

Warning:  More  than  1/5  of  Fitted  Cells  are  Sparse 
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iilili;  11;  M-  : 

value .  |  p||§|j 

PrOlM  :  =  : 

Pearson  Chi-Square 

3.383  2 

6.184 

Continuity  Adjusted  Chi-Square 

1.895  2 

0.388 

Likelihood  Ratio  Chi-Square 

3.499  2 

0.174 

|l|i;j;;pii;s|iCoenicient 

Contigency 

0.305 

Cramer's  V 

0.320 

Goodman-Kruskal  Gamma 

0.161 

0.275 

Kendalls  tau-b 

0.096 

0.166 

Stuart's  tau-c 

0.110 

0.190 

Somers  D  (C|R) 

0.110 

0.190 

Warning:  More  than  1/5  ot  Fitted  Cells  are  Sparse 
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